[Qemu-devel] [PATCH v2 1/1] memory.h: Improve IOMMU related documentation

[Peter Maydell]

Add more detail to the documentation for memory_region_init_iommu()
and other IOMMU-related functions and data structures.

Signed-off-by: Peter Maydell <***@linaro.org>
---
v1 -> v2 changes:
* documented replay method
* added note about wanting RCU or big qemu lock while calling
translate

include/exec/memory.h | 96 +++++++++++++++++++++++++++++++++++++++----
1 file changed, 87 insertions(+), 9 deletions(-)

diff --git a/include/exec/memory.h b/include/exec/memory.h
index 4402ba6c0d..997598c664 100644
--- a/include/exec/memory.h
+++ b/include/exec/memory.h
@@ -194,29 +194,95 @@ enum IOMMUMemoryRegionAttr {
IOMMU_ATTR_SPAPR_TCE_FD
};

+/**
+ * IOMMUMemoryRegionClass:
+ *
+ * All IOMMU implementations need to subclass TYPE_IOMMU_MEMORY_REGION
+ * and provide implementations of at least some of the methods here
+ * to handle requests to the memory region. The minimum requirement
+ * is a @translate method.
+ */
typedef struct IOMMUMemoryRegionClass {
/* private */
struct DeviceClass parent_class;

/*
* Return a TLB entry that contains a given address. Flag should
- * be the access permission of this translation operation. We can
- * set flag to IOMMU_NONE to mean that we don't need any
- * read/write permission checks, like, when for region replay.
+ * be the access permission of this translation operation. The
+ * flag may be set to IOMMU_NONE to mean that we don't need any
+ * read/write permission checks; this is used by the default
+ * implementation of memory_region_iommu_replay(). (See the documentation
+ * of the replay method below for more details.)
+ *
+ * Once the IOMMU has returned a TLB entry, it must notify
+ * the IOMMU's users if that TLB entry changes, using
+ * memory_region_notify_iommu() (or, if necessary, by calling
+ * memory_region_notify_one() for each registered notifier).
+ *
+ * The returned information remains valid while the caller is
+ * holding the big QEMU lock or is inside an RCU critical section;
+ * if the caller wishes to cache the mapping beyond that it must
+ * register an IOMMU notifier so it can invalidate its cached
+ * information when the IOMMU mapping changes..
+ *
+ * @iommu: the IOMMUMemoryRegion
+ * @hwaddr: address to be translated within the memory region
+ * @flag: requested access permissions
*/
IOMMUTLBEntry (*translate)(IOMMUMemoryRegion *iommu, hwaddr addr,
IOMMUAccessFlags flag);
- /* Returns minimum supported page size */
+ /* Returns minimum supported page size in bytes.
+ * If this method is not provided then the minimum is assumed to
+ * be TARGET_PAGE_SIZE.
+ *
+ * @iommu: the IOMMUMemoryRegion
+ */
uint64_t (*get_min_page_size)(IOMMUMemoryRegion *iommu);
- /* Called when IOMMU Notifier flag changed */
+ /* Called when IOMMU Notifier flag changes (ie when the set of
+ * events which IOMMU users are requesting notification for changes).
+ * Optional method -- need not be provided if the IOMMU does not
+ * need to know exactly which events must be notified.
+ *
+ * @iommu: the IOMMUMemoryRegion
+ * @old_flags: events which previously needed to be notified
+ * @new_flags: events which now need to be notified
+ */
void (*notify_flag_changed)(IOMMUMemoryRegion *iommu,
IOMMUNotifierFlag old_flags,
IOMMUNotifierFlag new_flags);
- /* Set this up to provide customized IOMMU replay function */
+ /* Called to handle memory_region_iommu_replay().
+ *
+ * The default implementation of memory_region_iommu_replay() is to
+ * call the IOMMU translate method for every page in the address space
+ * with flag == IOMMU_NONE and then call the notifier if translate
+ * returns valid mapping. If this method is implemented then it
+ * overrides the default behaviour, and must provide the full semantics
+ * of memory_region_iommu_replay(), by calling @notifier for every
+ * translation present in the IOMMU.
+ *
+ * Optional method -- an IOMMU only needs to provide this method
+ * if the default is inefficient or produces undesirable side effects.
+ *
+ * Note: this is not related to record-and-replay functionality.
+ */
void (*replay)(IOMMUMemoryRegion *iommu, IOMMUNotifier *notifier);

- /* Get IOMMU misc attributes */
- int (*get_attr)(IOMMUMemoryRegion *iommu, enum IOMMUMemoryRegionAttr,
+ /* Get IOMMU misc attributes. This is an optional method that
+ * can be used to allow users of the IOMMU to get implementation-specific
+ * information. The IOMMU implements this method to handle calls
+ * by IOMMU users to memory_region_iommu_get_attr() by filling in
+ * the arbitrary data pointer for any IOMMUMemoryRegionAttr values that
+ * the IOMMU supports. If the method is unimplemented then
+ * memory_region_iommu_get_attr() will always return -EINVAL.
+ *
+ * @iommu: the IOMMUMemoryRegion
+ * @attr: attribute being queried
+ * @data: memory to fill in with the attribute data
+ *
+ * Returns 0 on success, or a negative errno; in particular
+ * returns -EINVAL for unrecognized or unimplemented attribute types.
+ */
+ int (*get_attr)(IOMMUMemoryRegion *iommu, enum IOMMUMemoryRegionAttr attr,
void *data);
} IOMMUMemoryRegionClass;

@@ -705,6 +771,14 @@ static inline void memory_region_init_reservation(MemoryRegion *mr,
* An IOMMU region translates addresses and forwards accesses to a target
* memory region.
*
+ * The IOMMU implementation must define a subclass of TYPE_IOMMU_MEMORY_REGION.
+ * @_iommu_mr should be a pointer to enough memory for an instance of
+ * that subclass, @instance_size is the size of that subclass, and
+ * @mrtypename is its name. This function will initialize @_iommu_mr as an
+ * instance of the subclass, and its methods will then be called to handle
+ * accesses to the memory region. See the documentation of
+ * #IOMMUMemoryRegionClass for further details.
+ *
* @_iommu_mr: the #IOMMUMemoryRegion to be initialized
* @instance_size: the IOMMUMemoryRegion subclass instance size
* @mrtypename: the type name of the #IOMMUMemoryRegion
@@ -953,6 +1027,8 @@ void memory_region_register_iommu_notifier(MemoryRegion *mr,
* a notifier with the minimum page granularity returned by
* mr->iommu_ops->get_page_size().
*
+ * Note: this is not related to record-and-replay functionality.
+ *
* @iommu_mr: the memory region to observe
* @n: the notifier to which to replay iommu mappings
*/
@@ -962,6 +1038,8 @@ void memory_region_iommu_replay(IOMMUMemoryRegion *iommu_mr, IOMMUNotifier *n);
* memory_region_iommu_replay_all: replay existing IOMMU translations
* to all the notifiers registered.
*
+ * Note: this is not related to record-and-replay functionality.
+ *
* @iommu_mr: the memory region to observe
*/
void memory_region_iommu_replay_all(IOMMUMemoryRegion *iommu_mr);
@@ -981,7 +1059,7 @@ void memory_region_unregister_iommu_notifier(MemoryRegion *mr,
* memory_region_iommu_get_attr: return an IOMMU attr if get_attr() is
* defined on the IOMMU.
*
- * Returns 0 if succeded, error code otherwise.
+ * Returns 0 on success, or a negative errno otherwise.
*
* @iommu_mr: the memory region
* @attr: the requested attribute

[Auger Eric]

Hi Peter,

Post by Peter Maydell
Add more detail to the documentation for memory_region_init_iommu()
and other IOMMU-related functions and data structures.
---
* documented replay method
* added note about wanting RCU or big qemu lock while calling
translate
include/exec/memory.h | 96 +++++++++++++++++++++++++++++++++++++++----
1 file changed, 87 insertions(+), 9 deletions(-)
diff --git a/include/exec/memory.h b/include/exec/memory.h
index 4402ba6c0d..997598c664 100644
--- a/include/exec/memory.h
+++ b/include/exec/memory.h
@@ -194,29 +194,95 @@ enum IOMMUMemoryRegionAttr {
IOMMU_ATTR_SPAPR_TCE_FD
};
+/**
+ *
+ * All IOMMU implementations need to subclass TYPE_IOMMU_MEMORY_REGION
+ * and provide implementations of at least some of the methods here
+ * to handle requests to the memory region.

of at least the @translate method. Others are optional?
The minimum requirement

Post by Peter Maydell
+ */
typedef struct IOMMUMemoryRegionClass {
/* private */
struct DeviceClass parent_class;
/*
* Return a TLB entry that contains a given address. Flag should
- * be the access permission of this translation operation. We can
- * set flag to IOMMU_NONE to mean that we don't need any
- * read/write permission checks, like, when for region replay.
+ * be the access permission of this translation operation. The
+ * flag may be set to IOMMU_NONE to mean that we don't need any
+ * read/write permission checks; this is used by the default
+ * implementation of memory_region_iommu_replay(). (See the documentation
+ * of the replay method below for more details.)
+ *
+ * Once the IOMMU has returned a TLB entry, it must notify
+ * the IOMMU's users if that TLB entry changes, using
+ * memory_region_notify_iommu() (or, if necessary, by calling
+ * memory_region_notify_one() for each registered notifier).

if caching mode is set (VFIO integration), On the first guest map
(invalidation on map), the IOMMU must replay the mapping (notify VFIO)
to get the 1st stage of the physical IOMMU programmed. At that moment
the IOMMU may not have returned any TLB entry.

Maybe it is worth to define the term "IOMMU user", ie. someone who
registered an IOMMU notifier? In case the IOMMU is used without
vhost/VFIO, there is no registered notifier, however the IOMMU is "used".

Post by Peter Maydell
+ *
+ * The returned information remains valid while the caller is
+ * holding the big QEMU lock or is inside an RCU critical section;
+ * if the caller wishes to cache the mapping beyond that it must
+ * register an IOMMU notifier so it can invalidate its cached
+ * information when the IOMMU mapping changes..
+ *
*/
IOMMUTLBEntry (*translate)(IOMMUMemoryRegion *iommu, hwaddr addr,
IOMMUAccessFlags flag);
- /* Returns minimum supported page size */
+ /* Returns minimum supported page size in bytes.
+ * If this method is not provided then the minimum is assumed to
+ * be TARGET_PAGE_SIZE.
+ *
+ */
uint64_t (*get_min_page_size)(IOMMUMemoryRegion *iommu);
- /* Called when IOMMU Notifier flag changed */
+ /* Called when IOMMU Notifier flag changes (ie when the set of
+ * events which IOMMU users are requesting notification for changes).
+ * Optional method -- need not be provided if the IOMMU does not
+ * need to know exactly which events must be notified.
+ *
+ */
void (*notify_flag_changed)(IOMMUMemoryRegion *iommu,
IOMMUNotifierFlag old_flags,
IOMMUNotifierFlag new_flags);
- /* Set this up to provide customized IOMMU replay function */
+ /* Called to handle memory_region_iommu_replay().
+ *
+ * The default implementation of memory_region_iommu_replay() is to
+ * call the IOMMU translate method for every page in the address space
+ * with flag == IOMMU_NONE

generally meaning invalidation?
and then call the notifier if translate
I think possibly we could have several notifiers?

Post by Peter Maydell
+ * returns valid mapping

a valid mapping?
. If this method is implemented then it

Post by Peter Maydell
+ * overrides the default behaviour, and must provide the full semantics
+ * translation present in the IOMMU.

by first invalidating the whole range and then calling @notifier for
every valid mapping?

Post by Peter Maydell
+ *
+ * Optional method -- an IOMMU only needs to provide this method
+ * if the default is inefficient or produces undesirable side effects.
+ *
+ * Note: this is not related to record-and-replay functionality.

The record-and-replay functionality is not obvious to me.

Post by Peter Maydell
+ */
void (*replay)(IOMMUMemoryRegion *iommu, IOMMUNotifier *notifier);
- /* Get IOMMU misc attributes */
- int (*get_attr)(IOMMUMemoryRegion *iommu, enum IOMMUMemoryRegionAttr,
+ /* Get IOMMU misc attributes. This is an optional method that
+ * can be used to allow users of the IOMMU to get implementation-specific
+ * information. The IOMMU implements this method to handle calls
+ * by IOMMU users to memory_region_iommu_get_attr() by filling in
+ * the arbitrary data pointer for any IOMMUMemoryRegionAttr values that
+ * the IOMMU supports. If the method is unimplemented then
+ * memory_region_iommu_get_attr() will always return -EINVAL.
+ *
+ *
+ * Returns 0 on success, or a negative errno; in particular
+ * returns -EINVAL for unrecognized or unimplemented attribute types.
+ */
+ int (*get_attr)(IOMMUMemoryRegion *iommu, enum IOMMUMemoryRegionAttr attr,
void *data);
} IOMMUMemoryRegionClass;
@@ -705,6 +771,14 @@ static inline void memory_region_init_reservation(MemoryRegion *mr,
* An IOMMU region translates addresses and forwards accesses to a target
* memory region.
*
+ * The IOMMU implementation must define a subclass of TYPE_IOMMU_MEMORY_REGION.
+ * instance of the subclass, and its methods will then be called to handle
+ * accesses to the memory region. See the documentation of
+ * #IOMMUMemoryRegionClass for further details.
+ *
@@ -953,6 +1027,8 @@ void memory_region_register_iommu_notifier(MemoryRegion *mr,
* a notifier with the minimum page granularity returned by
* mr->iommu_ops->get_page_size().
*
+ * Note: this is not related to record-and-replay functionality.
+ *
*/
@@ -962,6 +1038,8 @@ void memory_region_iommu_replay(IOMMUMemoryRegion *iommu_mr, IOMMUNotifier *n);
* memory_region_iommu_replay_all: replay existing IOMMU translations
* to all the notifiers registered.
*
+ * Note: this is not related to record-and-replay functionality.
+ *
*/
void memory_region_iommu_replay_all(IOMMUMemoryRegion *iommu_mr);
@@ -981,7 +1059,7 @@ void memory_region_unregister_iommu_notifier(MemoryRegion *mr,
* memory_region_iommu_get_attr: return an IOMMU attr if get_attr() is
* defined on the IOMMU.
*
- * Returns 0 if succeded, error code otherwise.
+ * Returns 0 on success, or a negative errno otherwise.
*

Thanks

Eric

[Peter Maydell]

Post by Auger Eric
Hi Peter,

Post by Peter Maydell
Add more detail to the documentation for memory_region_init_iommu()
and other IOMMU-related functions and data structures.
---
* documented replay method
* added note about wanting RCU or big qemu lock while calling
translate

Hi; thanks for the review. I have some followup questions where
my understanding of the IOMMU is weak...

Post by Auger Eric

Post by Peter Maydell
include/exec/memory.h | 96 +++++++++++++++++++++++++++++++++++++++----
1 file changed, 87 insertions(+), 9 deletions(-)
diff --git a/include/exec/memory.h b/include/exec/memory.h
index 4402ba6c0d..997598c664 100644
--- a/include/exec/memory.h
+++ b/include/exec/memory.h
@@ -194,29 +194,95 @@ enum IOMMUMemoryRegionAttr {
IOMMU_ATTR_SPAPR_TCE_FD
};
+/**
+ *
+ * All IOMMU implementations need to subclass TYPE_IOMMU_MEMORY_REGION
+ * and provide implementations of at least some of the methods here
+ * to handle requests to the memory region.

Yes, that's probably a bit smoother wording.

Post by Auger Eric
The minimum requirement

Post by Peter Maydell
+ */
typedef struct IOMMUMemoryRegionClass {
/* private */
struct DeviceClass parent_class;
/*
* Return a TLB entry that contains a given address. Flag should
- * be the access permission of this translation operation. We can
- * set flag to IOMMU_NONE to mean that we don't need any
- * read/write permission checks, like, when for region replay.
+ * be the access permission of this translation operation. The
+ * flag may be set to IOMMU_NONE to mean that we don't need any
+ * read/write permission checks; this is used by the default
+ * implementation of memory_region_iommu_replay(). (See the documentation
+ * of the replay method below for more details.)
+ *
+ * Once the IOMMU has returned a TLB entry, it must notify
+ * the IOMMU's users if that TLB entry changes, using
+ * memory_region_notify_iommu() (or, if necessary, by calling
+ * memory_region_notify_one() for each registered notifier).

if caching mode is set (VFIO integration), On the first guest map
(invalidation on map), the IOMMU must replay the mapping (notify VFIO)
to get the 1st stage of the physical IOMMU programmed. At that moment
the IOMMU may not have returned any TLB entry.

I'm afraid you've lost me. What is caching mode, how do you set it,
how does the IOMMU replay the mapping? More concretely, what change
needs to be made to this text?

Post by Auger Eric
Maybe it is worth to define the term "IOMMU user", ie. someone who
registered an IOMMU notifier? In case the IOMMU is used without
vhost/VFIO, there is no registered notifier, however the IOMMU is "used".

I was using "IOMMU user" here in a fairly loose sense to mean anybody
using these interfaces (ie the code "outside" of the IOMMU implementation).

Maybe we should fix this by removing this paragraph and putting something
in the introductory "All IOMMU implementations need ..." instead as a
general description of the IOMMU's responsibilities regarding notifiers.

Post by Auger Eric

Post by Peter Maydell
+ *
+ * The returned information remains valid while the caller is
+ * holding the big QEMU lock or is inside an RCU critical section;
+ * if the caller wishes to cache the mapping beyond that it must
+ * register an IOMMU notifier so it can invalidate its cached
+ * information when the IOMMU mapping changes..
+ *
*/
IOMMUTLBEntry (*translate)(IOMMUMemoryRegion *iommu, hwaddr addr,
IOMMUAccessFlags flag);
- /* Returns minimum supported page size */
+ /* Returns minimum supported page size in bytes.
+ * If this method is not provided then the minimum is assumed to
+ * be TARGET_PAGE_SIZE.
+ *
+ */
uint64_t (*get_min_page_size)(IOMMUMemoryRegion *iommu);
- /* Called when IOMMU Notifier flag changed */
+ /* Called when IOMMU Notifier flag changes (ie when the set of
+ * events which IOMMU users are requesting notification for changes).
+ * Optional method -- need not be provided if the IOMMU does not
+ * need to know exactly which events must be notified.
+ *
+ */
void (*notify_flag_changed)(IOMMUMemoryRegion *iommu,
IOMMUNotifierFlag old_flags,
IOMMUNotifierFlag new_flags);
- /* Set this up to provide customized IOMMU replay function */
+ /* Called to handle memory_region_iommu_replay().
+ *
+ * The default implementation of memory_region_iommu_replay() is to
+ * call the IOMMU translate method for every page in the address space
+ * with flag == IOMMU_NONE

generally meaning invalidation?

The flags argument is something I find a bit confusing overall --
what does it mean to put a transaction through an IOMMU that
isn't either a read transaction or a write transaction? Do we
use IOMMU_NONE for anything except this special purpose "I am
the default replay implementation and just want to cause notifiers
to be called" ?

Post by Auger Eric
and then call the notifier if translate
I think possibly we could have several notifiers?

This method takes one notifier pointer and is responsible
for notifying it and only it, I think.

Post by Auger Eric

Post by Peter Maydell
+ * returns valid mapping

a valid mapping?

Yes.

Post by Auger Eric
. If this method is implemented then it

Post by Peter Maydell
+ * overrides the default behaviour, and must provide the full semantics
+ * translation present in the IOMMU.

every valid mapping?

Wait, why do we have to invalidate anything? I thought replay was
purely a "tell the notifier about the current state of the IOMMU"
function?

Post by Auger Eric

Post by Peter Maydell
+ *
+ * Optional method -- an IOMMU only needs to provide this method
+ * if the default is inefficient or produces undesirable side effects.
+ *
+ * Note: this is not related to record-and-replay functionality.

The record-and-replay functionality is not obvious to me.

This is just a note to say "this is not anything to do with what the
rest of the codebase calls 'replay', it's just an in-retrospect
poorly chosen method name, so don't confuse the two".

Post by Auger Eric

Post by Peter Maydell
+ */
void (*replay)(IOMMUMemoryRegion *iommu, IOMMUNotifier *notifier);

thanks
-- PMM

[Auger Eric]

Hi Peter,

Post by Peter Maydell

Post by Auger Eric
Hi Peter,

Post by Peter Maydell
Add more detail to the documentation for memory_region_init_iommu()
and other IOMMU-related functions and data structures.
---
* documented replay method
* added note about wanting RCU or big qemu lock while calling
translate

Hi; thanks for the review. I have some followup questions where
my understanding of the IOMMU is weak...

Post by Auger Eric

Post by Peter Maydell
include/exec/memory.h | 96 +++++++++++++++++++++++++++++++++++++++----
1 file changed, 87 insertions(+), 9 deletions(-)
diff --git a/include/exec/memory.h b/include/exec/memory.h
index 4402ba6c0d..997598c664 100644
--- a/include/exec/memory.h
+++ b/include/exec/memory.h
@@ -194,29 +194,95 @@ enum IOMMUMemoryRegionAttr {
IOMMU_ATTR_SPAPR_TCE_FD
};
+/**
+ *
+ * All IOMMU implementations need to subclass TYPE_IOMMU_MEMORY_REGION
+ * and provide implementations of at least some of the methods here
+ * to handle requests to the memory region.

Yes, that's probably a bit smoother wording.

Post by Auger Eric
The minimum requirement

Post by Peter Maydell
+ */
typedef struct IOMMUMemoryRegionClass {
/* private */
struct DeviceClass parent_class;
/*
* Return a TLB entry that contains a given address. Flag should
- * be the access permission of this translation operation. We can
- * set flag to IOMMU_NONE to mean that we don't need any
- * read/write permission checks, like, when for region replay.
+ * be the access permission of this translation operation. The
+ * flag may be set to IOMMU_NONE to mean that we don't need any
+ * read/write permission checks; this is used by the default
+ * implementation of memory_region_iommu_replay(). (See the documentation
+ * of the replay method below for more details.)
+ *
+ * Once the IOMMU has returned a TLB entry, it must notify
+ * the IOMMU's users if that TLB entry changes, using
+ * memory_region_notify_iommu() (or, if necessary, by calling
+ * memory_region_notify_one() for each registered notifier).

if caching mode is set (VFIO integration), On the first guest map
(invalidation on map), the IOMMU must replay the mapping (notify VFIO)
to get the 1st stage of the physical IOMMU programmed. At that moment
the IOMMU may not have returned any TLB entry.

I'm afraid you've lost me. What is caching mode, how do you set it,
how does the IOMMU replay the mapping? More concretely, what change
needs to be made to this text?

np. The caching mode relates to Intel IOMMU. When this bit is advertised
in the Intel IOMMU, it forces the IOMMU driver to invalidate TLB entries
also on map and not only on unmap. So when any iova is mapped, an
invalidation command is sent and this allows to trap when entries are
created. Then the IOMMU device can notify "users" that a mapping
happened and this mapping can be forwarded to the physical IOMMU.
Caching mode is set when a VFIO device is behind the IOMMU and both
stage 1 and stage 2 are combined in the physical IOMMU.

what about: the IOMMU is responsible to call the registered user
notifiers when the associated event occurs on an IOTLB entry (map/unmap).

Post by Peter Maydell

Post by Auger Eric
Maybe it is worth to define the term "IOMMU user", ie. someone who
registered an IOMMU notifier? In case the IOMMU is used without
vhost/VFIO, there is no registered notifier, however the IOMMU is "used".

I was using "IOMMU user" here in a fairly loose sense to mean anybody
using these interfaces (ie the code "outside" of the IOMMU implementation).
Maybe we should fix this by removing this paragraph and putting something
in the introductory "All IOMMU implementations need ..." instead as a
general description of the IOMMU's responsibilities regarding notifiers.

Post by Auger Eric

Post by Peter Maydell
+ *
+ * The returned information remains valid while the caller is
+ * holding the big QEMU lock or is inside an RCU critical section;
+ * if the caller wishes to cache the mapping beyond that it must
+ * register an IOMMU notifier so it can invalidate its cached
+ * information when the IOMMU mapping changes..
+ *
*/
IOMMUTLBEntry (*translate)(IOMMUMemoryRegion *iommu, hwaddr addr,
IOMMUAccessFlags flag);
- /* Returns minimum supported page size */
+ /* Returns minimum supported page size in bytes.
+ * If this method is not provided then the minimum is assumed to
+ * be TARGET_PAGE_SIZE.
+ *
+ */
uint64_t (*get_min_page_size)(IOMMUMemoryRegion *iommu);
- /* Called when IOMMU Notifier flag changed */
+ /* Called when IOMMU Notifier flag changes (ie when the set of
+ * events which IOMMU users are requesting notification for changes).
+ * Optional method -- need not be provided if the IOMMU does not
+ * need to know exactly which events must be notified.
+ *
+ */
void (*notify_flag_changed)(IOMMUMemoryRegion *iommu,
IOMMUNotifierFlag old_flags,
IOMMUNotifierFlag new_flags);
- /* Set this up to provide customized IOMMU replay function */
+ /* Called to handle memory_region_iommu_replay().
+ *
+ * The default implementation of memory_region_iommu_replay() is to
+ * call the IOMMU translate method for every page in the address space
+ * with flag == IOMMU_NONE

generally meaning invalidation?

The flags argument is something I find a bit confusing overall --
what does it mean to put a transaction through an IOMMU that
isn't either a read transaction or a write transaction? Do we
use IOMMU_NONE for anything except this special purpose "I am
the default replay implementation and just want to cause notifiers
to be called" ?

No I don't think so. To me the purpose of the change was for replay
(before we only had is_write and it was not obvious what it meant for a
replay action, hence the change I think).
bf55b7a memory: tune last param of iommu_ops.translate()

Post by Peter Maydell

Post by Auger Eric
and then call the notifier if translate
I think possibly we could have several notifiers?

This method takes one notifier pointer and is responsible
for notifying it and only it, I think.

oh yes forget it.

Post by Peter Maydell

Post by Auger Eric

Post by Peter Maydell
+ * returns valid mapping

a valid mapping?

Yes.

Post by Auger Eric
. If this method is implemented then it

Post by Peter Maydell
+ * overrides the default behaviour, and must provide the full semantics
+ * translation present in the IOMMU.

every valid mapping?

Wait, why do we have to invalidate anything? I thought replay was
purely a "tell the notifier about the current state of the IOMMU"
function?

Hum effectively the generic implementation of memory_region_iommu_replay
only notifies when a mapping exists.

Intel's custom method also removes stale entries and make the cache
consistent with page tables for the mr's span. This is also what I did
when I implemented the tlbi on map trick in smmuv3.

see hw/i386/intel_iommu.c, vtd_iommu_replay
/*
* The replay can be triggered by either a invalidation or a newly
* created entry. No matter what, we release existing mappings
* (it means flushing caches for UNMAP-only registers).
*/

However most probably that's due to the fact the function is also used
internally by the intel iommu on invalidations (context).

So you are right invalidation is not mandated.

Post by Peter Maydell

Post by Auger Eric

Post by Peter Maydell
+ *
+ * Optional method -- an IOMMU only needs to provide this method
+ * if the default is inefficient or produces undesirable side effects.
+ *
+ * Note: this is not related to record-and-replay functionality.

The record-and-replay functionality is not obvious to me.

This is just a note to say "this is not anything to do with what the
rest of the codebase calls 'replay', it's just an in-retrospect
poorly chosen method name, so don't confuse the two".

ok

Thanks

Eric

Post by Peter Maydell

Post by Auger Eric

Post by Peter Maydell
+ */
void (*replay)(IOMMUMemoryRegion *iommu, IOMMUNotifier *notifier);

thanks
-- PMM

[Auger Eric]

This post might be inappropriate. Click to display it.

[Peter Maydell]

Post by Auger Eric

This runs into something I found when I was implementing the Arm
Memory Protection Controller -- at the point when the guest changes
the config,

do you mean the config structures (STE, CD) or the page table update?

The Memory Protection Controller is not the SMMUv3. The MPC
config is set using some registers to write to the MPC's LUT
(lookup table).

Post by Auger Eric
it doesn't have enough information to be able to call a

"map" notifier, because the mapping depends on the memory transaction
attributes, it's not fixed and dependent only on the address.

I am not sure I understand what you mean here. When the notifier get's
called, aren't we supposed to look for the info in the actual page table
(where memory access control attributes and others can be found at that
time, ie. ARM AP[] for instance) and send those through the notifier (as
stored in the IOTLBEnry)?

The problem is that if your translations depend on the tx attributes,
ie "secure access to address X should translate to Y, but nonsecure
access to address X should translate to Z", then the notifier
API doesn't let you report that, because all it knows about is
unmap events which are "address X unmapped" and map events which
are "address X translates to Y".

Paolo has suggested some API changes in another thread (roughly,
having the notifier say which tx attributes matter for the translation,
and send multiple map events with appropriate information).

Post by Auger Eric
For instance in the intel iommu code, the whole table is parsed and the
replay hook is called for all valid entries.

This works because the intel iommu does not care about memory
transaction attributes: it has one translation for the input
address, regardless.

thanks
-- PMM