On Mon, May 25, 2020 at 12:05 PM Michael Walle michael@walle.cc wrote:
Am 2020-05-25 20:39, schrieb Saravana Kannan:
On Mon, May 25, 2020 at 4:31 AM Michael Walle michael@walle.cc wrote:
Am 2020-05-23 00:47, schrieb Michael Walle:
Am 2020-05-23 00:21, schrieb Saravana Kannan:
On Fri, May 22, 2020 at 11:41 AM Michael Walle michael@walle.cc wrote:
Am Mon, 18 May 2020 23:30:00 -0700 schrieb Saravana Kannan saravanak@google.com:
> When SYNC_STATE_ONLY support was added in commit 05ef983e0d65 ("driver > core: Add device link support for SYNC_STATE_ONLY flag"), > device_link_add() incorrectly skipped adding the new SYNC_STATE_ONLY > device link to the supplier's and consumer's "device link" list. > > This causes multiple issues: > - The device link is lost forever from driver core if the caller > didn't keep track of it (caller typically isn't expected to). This > is a memory leak. > - The device link is also never visible to any other code path after > device_link_add() returns. > > If we fix the "device link" list handling, that exposes a bunch of > issues. > > 1. The device link "status" state management code rightfully doesn't > handle the case where a DL_FLAG_MANAGED device link exists between a > supplier and consumer, but the consumer manages to probe successfully > before the supplier. The addition of DL_FLAG_SYNC_STATE_ONLY links > break this assumption. This causes device_links_driver_bound() to > throw a warning when this happens. > > Since DL_FLAG_SYNC_STATE_ONLY device links are mainly used for > creating proxy device links for child device dependencies and aren't > useful once the consumer device probes successfully, this patch just > deletes DL_FLAG_SYNC_STATE_ONLY device links once its consumer device > probes. This way, we avoid the warning, free up some memory and avoid > complicating the device links "status" state management code. > > 2. Creating a DL_FLAG_STATELESS device link between two devices that > already have a DL_FLAG_SYNC_STATE_ONLY device link will result in the > DL_FLAG_STATELESS flag not getting set correctly. This patch also > fixes this. > > Lastly, this patch also fixes minor whitespace issues.
My board triggers the WARN_ON(link->status != DL_STATE_CONSUMER_PROBE);
Full bootlog:
[..]
Thanks for the log and report. I haven't spent too much time thinking about this, but can you give this a shot? https://lore.kernel.org/lkml/20200520043626.181820-1-saravanak@google.com/
I've already tried that, as this is already in linux-next. Doesn't fix it, though.
btw. this only happens on linux-next (tested with next-20200522), not on 5.7-rc7 (which has the same two patches of yours)
I wouldn't be surprised if the difference is due to fw_devlink_pause/resume() calls in driver/of/property.c. It chops off ~1s in boot time by changing the order in which device links are created from DT. So, I think it's just masking the issue.
On linux-next where you see the issue, can you get the logs with this change: +++ b/drivers/base/core.c @@ -907,7 +907,10 @@ void device_links_driver_bound(struct device *dev) */ device_link_drop_managed(link); } else {
WARN_ON(link->status !=DL_STATE_CONSUMER_PROBE);
WARN(link->status != DL_STATE_CONSUMER_PROBE,"sup:%s - con:%s f:%d s:%d\n",dev_name(supplier),dev_name(link->consumer),
link->flags, link->status); WRITE_ONCE(link->status, DL_STATE_ACTIVE); }My goal is to figure out the order in which the device links between the supplier and consumers devices are created and how that's changing the flag and status. Then I can come up with a fix.
Here we go (hopefully, my mail client won't screw up the line wrapping):
Thanks for the logs!
Ok, that definitely gave me some more info. 1. It's happening only for this iommu which in some cases can create device links before fw_devlink through the use of BUS_NOTIFY_ADD_DEVICE. 2. The issue doesn't seem to be between STATELESS and SYNC_STATE_ONLY flags (because STATELESS flag is not set). 3. There seems to be a MANAGED link created by arm-smmu.c before/after the SYNC_STATE_ONLY link is created.
In which case, the SYNC_STATE_ONLY link is supposed to be a NOP, but that doesn't seem to be the case for some reason.
Can you add these debug messages and give me the logs? Hopefully this'll be my last log request. I tried reproducing this in hardware I have, but I couldn't reproduce it.
--- a/drivers/base/core.c +++ b/drivers/base/core.c @@ -348,6 +348,10 @@ struct device_link *device_link_add(struct device *consumer, if (flags & DL_FLAG_AUTOREMOVE_SUPPLIER) flags &= ~DL_FLAG_AUTOREMOVE_CONSUMER;
+ if (strstr(dev_name(supplier), "5000000.iommu")) { + dev_info(consumer, "Link attempted to %s 0x%x\n", dev_name(supplier), flags); + } + list_for_each_entry(link, &supplier->links.consumers, s_node) { if (link->consumer != consumer) continue; @@ -460,6 +464,10 @@ struct device_link *device_link_add(struct device *consumer, dev_dbg(consumer, "Linked as a consumer to %s\n", dev_name(supplier));
out: + if (strstr(dev_name(supplier), "5000000.iommu") && link) { + dev_info(consumer, "Link done to %s 0x%x %d\n", dev_name(supplier), link->flags, link->status); + } + device_pm_unlock(); device_links_write_unlock();
Thanks, Saravana