git-lng-odp

git-lng-odp@lists.linaro.org

1 participants
1249 discussions

[COMMIT] branch api-next updated. v1.15.0.0-421-g698d82ce

by git＠git-us.linaro.org

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, api-next has been updated via 698d82ce41a3f6ca08c6f218784d3eea6ac81e42 (commit) from cd097450cbf9d7bb6cd24654f8d43ceade5b114a (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 698d82ce41a3f6ca08c6f218784d3eea6ac81e42 Author: Nikhil Agarwal <nikhil.agarwal(a)linaro.org> Date: Thu Jul 13 12:00:06 2017 +0300 api: ipsec: add retain header capability Signed-off-by: Nikhil Agarwal <nikhil.agarwal(a)linaro.org> Reviewed-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-by: Balasubramanian Manoharan <bala.manoharan(a)linaro.org> Reviewed-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/include/odp/api/spec/ipsec.h b/include/odp/api/spec/ipsec.h index e602e4b8..54491b12 100644 --- a/include/odp/api/spec/ipsec.h +++ b/include/odp/api/spec/ipsec.h @@ -262,6 +262,12 @@ typedef struct odp_ipsec_capability_t { */ odp_support_t pipeline_cls; + /** + * Support of retaining outer headers (retain_outer) in inbound inline + * processed packets + */ + odp_support_t retain_header; + /** Maximum number of different destination CoSes in classification * pipelining. The same CoS may be used for many SAs. This is equal or * less than 'max_cos' capability in classifier API. ----------------------------------------------------------------------- Summary of changes: include/odp/api/spec/ipsec.h | 6 ++++++ 1 file changed, 6 insertions(+) hooks/post-receive --

9 years, 1 month

[COMMIT] branch api-next updated. v1.15.0.0-420-gcd097450

by git＠git-us.linaro.org

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, api-next has been updated via cd097450cbf9d7bb6cd24654f8d43ceade5b114a (commit) from ee5be324411a7520528a367967c28fc529d3bc2e (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit cd097450cbf9d7bb6cd24654f8d43ceade5b114a Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Mon Jun 26 13:24:28 2017 +0300 test: l2fwd: list not used features List not used features so that performance may be optimized on some platforms. E.g. on odp-linux implementation, timer polling causes high overhead, which is disabled when timers are not used. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/test/common_plat/performance/odp_l2fwd.c b/test/common_plat/performance/odp_l2fwd.c index 78b3633b..851614d6 100644 --- a/test/common_plat/performance/odp_l2fwd.c +++ b/test/common_plat/performance/odp_l2fwd.c @@ -1387,9 +1387,19 @@ int main(int argc, char *argv[]) odp_instance_t instance; int num_groups; odp_schedule_group_t group[MAX_PKTIOS]; + odp_init_t init; + + odp_init_param_init(&init); + + /* List features not to be used (may optimize performance) */ + init.not_used.feat.cls = 1; + init.not_used.feat.crypto = 1; + init.not_used.feat.ipsec = 1; + init.not_used.feat.timer = 1; + init.not_used.feat.tm = 1; /* Init ODP before calling anything else */ - if (odp_init_global(&instance, NULL, NULL)) { + if (odp_init_global(&instance, &init, NULL)) { LOG_ERR("Error: ODP global init failed.\n"); exit(EXIT_FAILURE); } ----------------------------------------------------------------------- Summary of changes: test/common_plat/performance/odp_l2fwd.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) hooks/post-receive --

9 years, 1 month

[COMMIT] branch master updated. v1.15.0.0-58-g490f4bf2

by git＠git-us.linaro.org

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 490f4bf22129638899ce71c99a8847e8ba849692 (commit) via 77f1b6743f97c6265e18cd47d0f597b44f71be2d (commit) via dcea50f7ad1f5473d74797e58eac5ba4abb939d1 (commit) from 7fc6d27e937b57b31360b07028388c811f8300dc (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 490f4bf22129638899ce71c99a8847e8ba849692 Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Thu Jul 13 15:58:28 2017 +0300 linux-gen: pool: avoid extra conversions Avoid conversions between buffer/packet handles and buffer/packet header pointers. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_pool_internal.h b/platform/linux-generic/include/odp_pool_internal.h index 1fb55d7d..a4c0e532 100644 --- a/platform/linux-generic/include/odp_pool_internal.h +++ b/platform/linux-generic/include/odp_pool_internal.h @@ -96,9 +96,8 @@ static inline odp_buffer_hdr_t *buf_hdl_to_hdr(odp_buffer_t buf) return (odp_buffer_hdr_t *)(uintptr_t)buf; } -int buffer_alloc_multi(pool_t *pool, odp_buffer_t buf[], - odp_buffer_hdr_t *buf_hdr[], int num); -void buffer_free_multi(const odp_buffer_t buf[], int num_free); +int buffer_alloc_multi(pool_t *pool, odp_buffer_hdr_t *buf_hdr[], int num); +void buffer_free_multi(odp_buffer_hdr_t *buf_hdr[], int num_free); #ifdef __cplusplus } diff --git a/platform/linux-generic/odp_packet.c b/platform/linux-generic/odp_packet.c index e6de2558..3cde2086 100644 --- a/platform/linux-generic/odp_packet.c +++ b/platform/linux-generic/odp_packet.c @@ -361,14 +361,14 @@ static inline void copy_num_segs(odp_packet_hdr_t *to, odp_packet_hdr_t *from, static inline odp_packet_hdr_t *alloc_segments(pool_t *pool, int num) { - odp_buffer_t buf[num]; odp_packet_hdr_t *pkt_hdr[num]; int ret; - ret = buffer_alloc_multi(pool, buf, (odp_buffer_hdr_t **)pkt_hdr, num); + ret = buffer_alloc_multi(pool, (odp_buffer_hdr_t **)pkt_hdr, num); + if (odp_unlikely(ret != num)) { if (ret > 0) - buffer_free_multi(buf, ret); + buffer_free_multi((odp_buffer_hdr_t **)pkt_hdr, ret); return NULL; } @@ -427,12 +427,12 @@ static inline odp_packet_hdr_t *add_segments(odp_packet_hdr_t *pkt_hdr, static inline void free_bufs(odp_packet_hdr_t *pkt_hdr, int first, int num) { int i; - odp_buffer_t buf[num]; + odp_buffer_hdr_t *buf_hdr[num]; for (i = 0; i < num; i++) - buf[i] = buffer_handle(pkt_hdr->buf_hdr.seg[first + i].hdr); + buf_hdr[i] = pkt_hdr->buf_hdr.seg[first + i].hdr; - buffer_free_multi(buf, num); + buffer_free_multi(buf_hdr, num); } static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, @@ -444,10 +444,10 @@ static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, if (head) { odp_packet_hdr_t *new_hdr; int i; - odp_buffer_t buf[num]; + odp_buffer_hdr_t *buf_hdr[num]; for (i = 0; i < num; i++) - buf[i] = buffer_handle(pkt_hdr->buf_hdr.seg[i].hdr); + buf_hdr[i] = pkt_hdr->buf_hdr.seg[i].hdr; /* First remaining segment is the new packet descriptor */ new_hdr = pkt_hdr->buf_hdr.seg[num].hdr; @@ -464,7 +464,7 @@ static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, pkt_hdr = new_hdr; - buffer_free_multi(buf, num); + buffer_free_multi(buf_hdr, num); } else { /* Free last 'num' bufs */ free_bufs(pkt_hdr, num_remain, num); @@ -487,10 +487,9 @@ static inline int packet_alloc(pool_t *pool, uint32_t len, int max_pkt, int num_buf, i; int num = max_pkt; int max_buf = max_pkt * num_seg; - odp_buffer_t buf[max_buf]; odp_packet_hdr_t *pkt_hdr[max_buf]; - num_buf = buffer_alloc_multi(pool, buf, (odp_buffer_hdr_t **)pkt_hdr, + num_buf = buffer_alloc_multi(pool, (odp_buffer_hdr_t **)pkt_hdr, max_buf); /* Failed to allocate all segments */ @@ -500,8 +499,12 @@ static inline int packet_alloc(pool_t *pool, uint32_t len, int max_pkt, num = num_buf / num_seg; num_free = num_buf - (num * num_seg); - if (num_free > 0) - buffer_free_multi(&buf[num_buf - num_free], num_free); + if (num_free > 0) { + odp_buffer_hdr_t **p; + + p = (odp_buffer_hdr_t **)&pkt_hdr[num_buf - num_free]; + buffer_free_multi(p, num_free); + } if (num == 0) return 0; @@ -584,43 +587,39 @@ void odp_packet_free(odp_packet_t pkt) int num_seg = pkt_hdr->buf_hdr.segcount; if (odp_likely(CONFIG_PACKET_MAX_SEGS == 1 || num_seg == 1)) - buffer_free_multi(&hdl, 1); + buffer_free_multi((odp_buffer_hdr_t **)&hdl, 1); else free_bufs(pkt_hdr, 0, num_seg); } void odp_packet_free_multi(const odp_packet_t pkt[], int num) { - odp_buffer_t buf[num * CONFIG_PACKET_MAX_SEGS]; - int i; - if (CONFIG_PACKET_MAX_SEGS == 1) { - for (i = 0; i < num; i++) - buf[i] = buffer_handle(packet_hdr(pkt[i])); - - buffer_free_multi(buf, num); + buffer_free_multi((odp_buffer_hdr_t **)(uintptr_t)pkt, num); } else { + odp_buffer_hdr_t *buf_hdr[num * CONFIG_PACKET_MAX_SEGS]; + int i; int j; int bufs = 0; for (i = 0; i < num; i++) { odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt[i]); int num_seg = pkt_hdr->buf_hdr.segcount; - odp_buffer_hdr_t *buf_hdr = &pkt_hdr->buf_hdr; + odp_buffer_hdr_t *hdr = &pkt_hdr->buf_hdr; - buf[bufs] = buffer_handle(pkt_hdr); + buf_hdr[bufs] = hdr; bufs++; if (odp_likely(num_seg == 1)) continue; for (j = 1; j < num_seg; j++) { - buf[bufs] = buffer_handle(buf_hdr->seg[j].hdr); + buf_hdr[bufs] = hdr->seg[j].hdr; bufs++; } } - buffer_free_multi(buf, bufs); + buffer_free_multi(buf_hdr, bufs); } } diff --git a/platform/linux-generic/odp_pool.c b/platform/linux-generic/odp_pool.c index 7e0c7606..e08c8985 100644 --- a/platform/linux-generic/odp_pool.c +++ b/platform/linux-generic/odp_pool.c @@ -73,13 +73,6 @@ static inline odp_buffer_hdr_t *buf_hdr_from_index(pool_t *pool, return buf_hdr; } -static inline uint32_t buf_index_from_hdl(odp_buffer_t buf) -{ - odp_buffer_hdr_t *buf_hdr = buf_hdl_to_hdr(buf); - - return buf_hdr->index; -} - int odp_pool_init_global(void) { uint32_t i; @@ -604,8 +597,7 @@ int odp_pool_info(odp_pool_t pool_hdl, odp_pool_info_t *info) return 0; } -int buffer_alloc_multi(pool_t *pool, odp_buffer_t buf[], - odp_buffer_hdr_t *buf_hdr_out[], int max_num) +int buffer_alloc_multi(pool_t *pool, odp_buffer_hdr_t *buf_hdr[], int max_num) { ring_t *ring; uint32_t mask, i; @@ -633,11 +625,7 @@ int buffer_alloc_multi(pool_t *pool, odp_buffer_t buf[], for (i = 0; i < num_ch; i++) { uint32_t j = cache_num - num_ch + i; - hdr = buf_hdr_from_index(pool, cache->buf_index[j]); - buf[i] = buf_from_buf_hdr(hdr); - - if (odp_likely(buf_hdr_out != NULL)) - buf_hdr_out[i] = hdr; + buf_hdr[i] = buf_hdr_from_index(pool, cache->buf_index[j]); } /* If needed, get more from the global pool */ @@ -661,11 +649,7 @@ int buffer_alloc_multi(pool_t *pool, odp_buffer_t buf[], hdr = buf_hdr_from_index(pool, data[i]); odp_prefetch(hdr); - - buf[idx] = buf_from_buf_hdr(hdr); - - if (odp_likely(buf_hdr_out != NULL)) - buf_hdr_out[idx] = hdr; + buf_hdr[idx] = hdr; } /* Cache extra buffers. Cache is currently empty. */ @@ -681,7 +665,7 @@ int buffer_alloc_multi(pool_t *pool, odp_buffer_t buf[], } static inline void buffer_free_to_pool(pool_t *pool, - const odp_buffer_t buf[], int num) + odp_buffer_hdr_t *buf_hdr[], int num) { int i; ring_t *ring; @@ -699,7 +683,7 @@ static inline void buffer_free_to_pool(pool_t *pool, ring = &pool->ring->hdr; mask = pool->ring_mask; for (i = 0; i < num; i++) - buf_index[i] = buf_index_from_hdl(buf[i]); + buf_index[i] = buf_hdr[i]->index; ring_enq_multi(ring, mask, buf_index, num); @@ -737,12 +721,12 @@ static inline void buffer_free_to_pool(pool_t *pool, } for (i = 0; i < num; i++) - cache->buf_index[cache_num + i] = buf_index_from_hdl(buf[i]); + cache->buf_index[cache_num + i] = buf_hdr[i]->index; cache->num = cache_num + num; } -void buffer_free_multi(const odp_buffer_t buf[], int num_total) +void buffer_free_multi(odp_buffer_hdr_t *buf_hdr[], int num_total) { pool_t *pool; int num; @@ -750,21 +734,20 @@ void buffer_free_multi(const odp_buffer_t buf[], int num_total) int first = 0; while (1) { - num = 1; - i = 1; - - pool = pool_from_buf(buf[first]); + num = 1; + i = 1; + pool = buf_hdr[first]->pool_ptr; /* 'num' buffers are from the same pool */ if (num_total > 1) { for (i = first; i < num_total; i++) - if (pool != pool_from_buf(buf[i])) + if (pool != buf_hdr[i]->pool_ptr) break; num = i - first; } - buffer_free_to_pool(pool, &buf[first], num); + buffer_free_to_pool(pool, &buf_hdr[first], num); if (i == num_total) return; @@ -782,7 +765,7 @@ odp_buffer_t odp_buffer_alloc(odp_pool_t pool_hdl) ODP_ASSERT(ODP_POOL_INVALID != pool_hdl); pool = pool_entry_from_hdl(pool_hdl); - ret = buffer_alloc_multi(pool, &buf, NULL, 1); + ret = buffer_alloc_multi(pool, (odp_buffer_hdr_t **)&buf, 1); if (odp_likely(ret == 1)) return buf; @@ -798,17 +781,17 @@ int odp_buffer_alloc_multi(odp_pool_t pool_hdl, odp_buffer_t buf[], int num) pool = pool_entry_from_hdl(pool_hdl); - return buffer_alloc_multi(pool, buf, NULL, num); + return buffer_alloc_multi(pool, (odp_buffer_hdr_t **)buf, num); } void odp_buffer_free(odp_buffer_t buf) { - buffer_free_multi(&buf, 1); + buffer_free_multi((odp_buffer_hdr_t **)&buf, 1); } void odp_buffer_free_multi(const odp_buffer_t buf[], int num) { - buffer_free_multi(buf, num); + buffer_free_multi((odp_buffer_hdr_t **)(uintptr_t)buf, num); } int odp_pool_capability(odp_pool_capability_t *capa) commit 77f1b6743f97c6265e18cd47d0f597b44f71be2d Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Thu Jul 13 15:58:27 2017 +0300 linux-gen: queue: avoid extra conversions Conversions are not needed as event handles are buffer header pointers. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_buffer_inlines.h b/platform/linux-generic/include/odp_buffer_inlines.h index 1dbc725b..a5658e81 100644 --- a/platform/linux-generic/include/odp_buffer_inlines.h +++ b/platform/linux-generic/include/odp_buffer_inlines.h @@ -28,11 +28,6 @@ static inline odp_buffer_t buf_from_buf_hdr(odp_buffer_hdr_t *hdr) return (odp_buffer_t)hdr; } -static inline odp_event_t event_from_buf_hdr(odp_buffer_hdr_t *hdr) -{ - return (odp_event_t)hdr; -} - #ifdef __cplusplus } #endif diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c index 348d609a..904c9ef2 100644 --- a/platform/linux-generic/odp_queue.c +++ b/platform/linux-generic/odp_queue.c @@ -422,43 +422,34 @@ static int queue_int_enq(queue_t q_int, odp_buffer_hdr_t *buf_hdr) static int queue_enq_multi(odp_queue_t handle, const odp_event_t ev[], int num) { - odp_buffer_hdr_t *buf_hdr[QUEUE_MULTI_MAX]; - queue_entry_t *queue; - int i; + queue_entry_t *queue = handle_to_qentry(handle); + + if (odp_unlikely(num == 0)) + return 0; if (num > QUEUE_MULTI_MAX) num = QUEUE_MULTI_MAX; - queue = handle_to_qentry(handle); - - for (i = 0; i < num; i++) - buf_hdr[i] = buf_hdl_to_hdr(odp_buffer_from_event(ev[i])); - - return num == 0 ? 0 : queue->s.enqueue_multi(qentry_to_int(queue), - buf_hdr, num); + return queue->s.enqueue_multi(qentry_to_int(queue), + (odp_buffer_hdr_t **)(uintptr_t)ev, num); } static int queue_enq(odp_queue_t handle, odp_event_t ev) { - odp_buffer_hdr_t *buf_hdr; - queue_entry_t *queue; - - queue = handle_to_qentry(handle); - buf_hdr = buf_hdl_to_hdr(odp_buffer_from_event(ev)); + queue_entry_t *queue = handle_to_qentry(handle); - return queue->s.enqueue(qentry_to_int(queue), buf_hdr); + return queue->s.enqueue(qentry_to_int(queue), + (odp_buffer_hdr_t *)(uintptr_t)ev); } -static inline int deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], +static inline int deq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], int num) { odp_buffer_hdr_t *hdr, *next; int i, j; - queue_entry_t *queue; int updated = 0; int status_sync = sched_fn->status_sync; - queue = qentry_from_int(q_int); LOCK(&queue->s.lock); if (odp_unlikely(queue->s.status < QUEUE_STATUS_READY)) { /* Bad queue, or queue has been destroyed. @@ -529,15 +520,18 @@ static inline int deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], static int queue_int_deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int num) { - return deq_multi(q_int, buf_hdr, num); + queue_entry_t *queue = qentry_from_int(q_int); + + return deq_multi(queue, buf_hdr, num); } static odp_buffer_hdr_t *queue_int_deq(queue_t q_int) { + queue_entry_t *queue = qentry_from_int(q_int); odp_buffer_hdr_t *buf_hdr = NULL; int ret; - ret = deq_multi(q_int, &buf_hdr, 1); + ret = deq_multi(queue, &buf_hdr, 1); if (ret == 1) return buf_hdr; @@ -545,38 +539,22 @@ static odp_buffer_hdr_t *queue_int_deq(queue_t q_int) return NULL; } -static int queue_deq_multi(odp_queue_t handle, odp_event_t events[], int num) +static int queue_deq_multi(odp_queue_t handle, odp_event_t ev[], int num) { - queue_entry_t *queue; - odp_buffer_hdr_t *buf_hdr[QUEUE_MULTI_MAX]; - int i, ret; + queue_entry_t *queue = handle_to_qentry(handle); if (num > QUEUE_MULTI_MAX) num = QUEUE_MULTI_MAX; - queue = handle_to_qentry(handle); - - ret = queue->s.dequeue_multi(qentry_to_int(queue), buf_hdr, num); - - for (i = 0; i < ret; i++) - events[i] = event_from_buf_hdr(buf_hdr[i]); - - return ret; + return queue->s.dequeue_multi(qentry_to_int(queue), + (odp_buffer_hdr_t **)ev, num); } - static odp_event_t queue_deq(odp_queue_t handle) { - queue_entry_t *queue; - odp_buffer_hdr_t *buf_hdr; - - queue = handle_to_qentry(handle); - buf_hdr = queue->s.dequeue(qentry_to_int(queue)); - - if (buf_hdr) - return event_from_buf_hdr(buf_hdr); + queue_entry_t *queue = handle_to_qentry(handle); - return ODP_EVENT_INVALID; + return (odp_event_t)queue->s.dequeue(qentry_to_int(queue)); } static int queue_init(queue_entry_t *queue, const char *name, @@ -678,17 +656,9 @@ odp_queue_t sched_cb_queue_handle(uint32_t queue_index) int sched_cb_queue_deq_multi(uint32_t queue_index, odp_event_t ev[], int num) { - int i, ret; queue_entry_t *qe = get_qentry(queue_index); - odp_buffer_hdr_t *buf_hdr[num]; - ret = deq_multi(qentry_to_int(qe), buf_hdr, num); - - if (ret > 0) - for (i = 0; i < ret; i++) - ev[i] = event_from_buf_hdr(buf_hdr[i]); - - return ret; + return deq_multi(qe, (odp_buffer_hdr_t **)ev, num); } int sched_cb_queue_empty(uint32_t queue_index) commit dcea50f7ad1f5473d74797e58eac5ba4abb939d1 Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Thu Jul 13 15:58:26 2017 +0300 linux-gen: buffer: change buffer handle to pointer Changed buffer, event and timeout handles to be pointers. Packet handles are already pointers. This enabled code optimization as some conversions may be removed and remaining once are just type casts. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/include/odp/arch/default/api/abi/buffer.h b/include/odp/arch/default/api/abi/buffer.h index eec6f01f..d8bfc913 100644 --- a/include/odp/arch/default/api/abi/buffer.h +++ b/include/odp/arch/default/api/abi/buffer.h @@ -14,18 +14,13 @@ extern "C" { /** @internal Dummy type for strong typing */ typedef struct { char dummy; /**< @internal Dummy */ } _odp_abi_buffer_t; -/** @internal Dummy type for strong typing */ -typedef struct { char dummy; /**< @internal Dummy */ } _odp_abi_buffer_seg_t; - /** @ingroup odp_buffer * @{ */ typedef _odp_abi_buffer_t *odp_buffer_t; -typedef _odp_abi_buffer_seg_t *odp_buffer_seg_t; -#define ODP_BUFFER_INVALID ((odp_buffer_t)0xffffffff) -#define ODP_SEGMENT_INVALID ((odp_buffer_seg_t)0xffffffff) +#define ODP_BUFFER_INVALID ((odp_buffer_t)NULL) /** * @} diff --git a/include/odp/arch/default/api/abi/event.h b/include/odp/arch/default/api/abi/event.h index 4f6596b1..fd86f25c 100644 --- a/include/odp/arch/default/api/abi/event.h +++ b/include/odp/arch/default/api/abi/event.h @@ -22,7 +22,7 @@ typedef struct { char dummy; /**< @internal Dummy */ } _odp_abi_event_t; typedef _odp_abi_event_t *odp_event_t; -#define ODP_EVENT_INVALID ((odp_event_t)0xffffffff) +#define ODP_EVENT_INVALID ((odp_event_t)NULL) typedef enum odp_event_type_t { ODP_EVENT_BUFFER = 1, diff --git a/platform/linux-generic/include/odp/api/plat/buffer_types.h b/platform/linux-generic/include/odp/api/plat/buffer_types.h index 809768f3..8b79bb52 100644 --- a/platform/linux-generic/include/odp/api/plat/buffer_types.h +++ b/platform/linux-generic/include/odp/api/plat/buffer_types.h @@ -31,11 +31,7 @@ extern "C" { typedef ODP_HANDLE_T(odp_buffer_t); -#define ODP_BUFFER_INVALID _odp_cast_scalar(odp_buffer_t, 0xffffffff) - -typedef ODP_HANDLE_T(odp_buffer_seg_t); - -#define ODP_SEGMENT_INVALID ((odp_buffer_seg_t)ODP_BUFFER_INVALID) +#define ODP_BUFFER_INVALID _odp_cast_scalar(odp_buffer_t, NULL) /** * @} diff --git a/platform/linux-generic/include/odp/api/plat/event_types.h b/platform/linux-generic/include/odp/api/plat/event_types.h index a1aa0e45..559a2fa0 100644 --- a/platform/linux-generic/include/odp/api/plat/event_types.h +++ b/platform/linux-generic/include/odp/api/plat/event_types.h @@ -32,7 +32,7 @@ extern "C" { typedef ODP_HANDLE_T(odp_event_t); -#define ODP_EVENT_INVALID _odp_cast_scalar(odp_event_t, 0xffffffff) +#define ODP_EVENT_INVALID _odp_cast_scalar(odp_event_t, NULL) typedef enum odp_event_type_t { ODP_EVENT_BUFFER = 1, diff --git a/platform/linux-generic/include/odp/api/plat/timer_types.h b/platform/linux-generic/include/odp/api/plat/timer_types.h index 8821bed6..a8891f11 100644 --- a/platform/linux-generic/include/odp/api/plat/timer_types.h +++ b/platform/linux-generic/include/odp/api/plat/timer_types.h @@ -38,7 +38,7 @@ typedef ODP_HANDLE_T(odp_timer_t); typedef ODP_HANDLE_T(odp_timeout_t); -#define ODP_TIMEOUT_INVALID _odp_cast_scalar(odp_timeout_t, 0xffffffff) +#define ODP_TIMEOUT_INVALID _odp_cast_scalar(odp_timeout_t, NULL) /** * @} diff --git a/platform/linux-generic/include/odp_buffer_inlines.h b/platform/linux-generic/include/odp_buffer_inlines.h index cf817d90..1dbc725b 100644 --- a/platform/linux-generic/include/odp_buffer_inlines.h +++ b/platform/linux-generic/include/odp_buffer_inlines.h @@ -23,9 +23,14 @@ odp_event_type_t _odp_buffer_event_type(odp_buffer_t buf); void _odp_buffer_event_type_set(odp_buffer_t buf, int ev); int odp_buffer_snprint(char *str, uint32_t n, odp_buffer_t buf); -static inline odp_buffer_t odp_hdr_to_buf(odp_buffer_hdr_t *hdr) +static inline odp_buffer_t buf_from_buf_hdr(odp_buffer_hdr_t *hdr) { - return hdr->handle.handle; + return (odp_buffer_t)hdr; +} + +static inline odp_event_t event_from_buf_hdr(odp_buffer_hdr_t *hdr) +{ + return (odp_event_t)hdr; } #ifdef __cplusplus diff --git a/platform/linux-generic/include/odp_buffer_internal.h b/platform/linux-generic/include/odp_buffer_internal.h index 076abe96..1e8a6486 100644 --- a/platform/linux-generic/include/odp_buffer_internal.h +++ b/platform/linux-generic/include/odp_buffer_internal.h @@ -33,25 +33,12 @@ extern "C" { #include <odp_schedule_if.h> #include <stddef.h> -typedef union odp_buffer_bits_t { - odp_buffer_t handle; - - union { - uint32_t u32; - - struct { - uint32_t pool_id: 8; - uint32_t index: 24; - }; - }; -} odp_buffer_bits_t; - #define BUFFER_BURST_SIZE CONFIG_BURST_SIZE /* Common buffer header */ struct odp_buffer_hdr_t { - /* Handle union */ - odp_buffer_bits_t handle; + /* Buffer index in the pool */ + uint32_t index; /* Initial buffer data pointer and length */ uint8_t *base_data; @@ -104,9 +91,13 @@ struct odp_buffer_hdr_t { * offset has to be used */ uint64_t ipc_data_offset; - /* Pool handle */ + /* Pool handle: will be removed, used only for odp_packet_pool() + * inlining */ odp_pool_t pool_hdl; + /* Pool pointer */ + void *pool_ptr; + /* Data or next header */ uint8_t data[0]; }; diff --git a/platform/linux-generic/include/odp_packet_internal.h b/platform/linux-generic/include/odp_packet_internal.h index d513f05e..d38f831f 100644 --- a/platform/linux-generic/include/odp_packet_internal.h +++ b/platform/linux-generic/include/odp_packet_internal.h @@ -208,11 +208,8 @@ int packet_parse_layer(odp_packet_hdr_t *pkt_hdr, /* Reset parser metadata for a new parse */ void packet_parse_reset(odp_packet_hdr_t *pkt_hdr); -/* Convert a packet handle to a buffer handle */ -odp_buffer_t _odp_packet_to_buffer(odp_packet_t pkt); - /* Convert a buffer handle to a packet handle */ -odp_packet_t _odp_packet_from_buffer(odp_buffer_t buf); +odp_packet_t _odp_packet_from_buf_hdr(odp_buffer_hdr_t *buf_hdr); static inline int packet_hdr_has_l2(odp_packet_hdr_t *pkt_hdr) { diff --git a/platform/linux-generic/include/odp_pool_internal.h b/platform/linux-generic/include/odp_pool_internal.h index ebb779da..1fb55d7d 100644 --- a/platform/linux-generic/include/odp_pool_internal.h +++ b/platform/linux-generic/include/odp_pool_internal.h @@ -28,8 +28,7 @@ extern "C" { typedef struct pool_cache_t { uint32_t num; - - odp_buffer_t buf[CONFIG_POOL_CACHE_SIZE]; + uint32_t buf_index[CONFIG_POOL_CACHE_SIZE]; } pool_cache_t ODP_ALIGNED_CACHE; @@ -92,34 +91,9 @@ static inline pool_t *pool_entry_from_hdl(odp_pool_t pool_hdl) return &pool_tbl->pool[_odp_typeval(pool_hdl)]; } -static inline odp_buffer_hdr_t *pool_buf_hdl_to_hdr(pool_t *pool, - odp_buffer_t buf) -{ - odp_buffer_bits_t handle; - uint32_t index, block_offset; - odp_buffer_hdr_t *buf_hdr; - - handle.handle = buf; - index = handle.index; - block_offset = index * pool->block_size; - - /* clang requires cast to uintptr_t */ - buf_hdr = (odp_buffer_hdr_t *)(uintptr_t)&pool->base_addr[block_offset]; - - return buf_hdr; -} - static inline odp_buffer_hdr_t *buf_hdl_to_hdr(odp_buffer_t buf) { - odp_buffer_bits_t handle; - uint32_t pool_id; - pool_t *pool; - - handle.handle = buf; - pool_id = handle.pool_id; - pool = pool_entry(pool_id); - - return pool_buf_hdl_to_hdr(pool, buf); + return (odp_buffer_hdr_t *)(uintptr_t)buf; } int buffer_alloc_multi(pool_t *pool, odp_buffer_t buf[], diff --git a/platform/linux-generic/odp_buffer.c b/platform/linux-generic/odp_buffer.c index 88c8140b..9c7dc1f5 100644 --- a/platform/linux-generic/odp_buffer.c +++ b/platform/linux-generic/odp_buffer.c @@ -41,6 +41,7 @@ uint32_t odp_buffer_size(odp_buffer_t buf) int odp_buffer_snprint(char *str, uint32_t n, odp_buffer_t buf) { odp_buffer_hdr_t *hdr; + pool_t *pool; int len = 0; if (!odp_buffer_is_valid(buf)) { @@ -49,12 +50,13 @@ int odp_buffer_snprint(char *str, uint32_t n, odp_buffer_t buf) } hdr = buf_hdl_to_hdr(buf); + pool = hdr->pool_ptr; len += snprintf(&str[len], n-len, "Buffer\n"); len += snprintf(&str[len], n-len, " pool %" PRIu64 "\n", - odp_pool_to_u64(hdr->pool_hdl)); + odp_pool_to_u64(pool->pool_hdl)); len += snprintf(&str[len], n-len, " addr %p\n", hdr->seg[0].data); len += snprintf(&str[len], n-len, diff --git a/platform/linux-generic/odp_packet.c b/platform/linux-generic/odp_packet.c index eb66af2d..e6de2558 100644 --- a/platform/linux-generic/odp_packet.c +++ b/platform/linux-generic/odp_packet.c @@ -54,7 +54,7 @@ static inline odp_packet_hdr_t *packet_hdr(odp_packet_t pkt) static inline odp_buffer_t buffer_handle(odp_packet_hdr_t *pkt_hdr) { - return pkt_hdr->buf_hdr.handle.handle; + return (odp_buffer_t)pkt_hdr; } static inline odp_packet_hdr_t *buf_to_packet_hdr(odp_buffer_t buf) @@ -62,6 +62,16 @@ static inline odp_packet_hdr_t *buf_to_packet_hdr(odp_buffer_t buf) return (odp_packet_hdr_t *)buf_hdl_to_hdr(buf); } +odp_packet_t _odp_packet_from_buf_hdr(odp_buffer_hdr_t *buf_hdr) +{ + return (odp_packet_t)buf_hdr; +} + +static inline odp_buffer_t packet_to_buffer(odp_packet_t pkt) +{ + return (odp_buffer_t)pkt; +} + static inline uint32_t packet_seg_len(odp_packet_hdr_t *pkt_hdr, uint32_t seg_idx) { @@ -617,7 +627,7 @@ void odp_packet_free_multi(const odp_packet_t pkt[], int num) int odp_packet_reset(odp_packet_t pkt, uint32_t len) { odp_packet_hdr_t *const pkt_hdr = packet_hdr(pkt); - pool_t *pool = pool_entry_from_hdl(pkt_hdr->buf_hdr.pool_hdl); + pool_t *pool = pkt_hdr->buf_hdr.pool_ptr; if (len > pool->headroom + pool->data_size + pool->tailroom) return -1; @@ -627,22 +637,6 @@ int odp_packet_reset(odp_packet_t pkt, uint32_t len) return 0; } -odp_packet_t _odp_packet_from_buffer(odp_buffer_t buf) -{ - if (odp_unlikely(buf == ODP_BUFFER_INVALID)) - return ODP_PACKET_INVALID; - - return (odp_packet_t)buf_to_packet_hdr(buf); -} - -odp_buffer_t _odp_packet_to_buffer(odp_packet_t pkt) -{ - if (odp_unlikely(pkt == ODP_PACKET_INVALID)) - return ODP_BUFFER_INVALID; - - return buffer_handle(packet_hdr(pkt)); -} - odp_packet_t odp_packet_from_event(odp_event_t ev) { if (odp_unlikely(ev == ODP_EVENT_INVALID)) @@ -874,7 +868,7 @@ int odp_packet_extend_head(odp_packet_t *pkt, uint32_t len, int ret = 0; if (len > headroom) { - pool_t *pool = pool_entry_from_hdl(pkt_hdr->buf_hdr.pool_hdl); + pool_t *pool = pkt_hdr->buf_hdr.pool_ptr; int num; int segs; @@ -1040,7 +1034,7 @@ int odp_packet_extend_tail(odp_packet_t *pkt, uint32_t len, int ret = 0; if (len > tailroom) { - pool_t *pool = pool_entry_from_hdl(pkt_hdr->buf_hdr.pool_hdl); + pool_t *pool = pkt_hdr->buf_hdr.pool_ptr; int num; int segs; @@ -1337,12 +1331,13 @@ int odp_packet_add_data(odp_packet_t *pkt_ptr, uint32_t offset, uint32_t len) odp_packet_t pkt = *pkt_ptr; odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); uint32_t pktlen = pkt_hdr->frame_len; + pool_t *pool = pkt_hdr->buf_hdr.pool_ptr; odp_packet_t newpkt; if (offset > pktlen) return -1; - newpkt = odp_packet_alloc(pkt_hdr->buf_hdr.pool_hdl, pktlen + len); + newpkt = odp_packet_alloc(pool->pool_hdl, pktlen + len); if (newpkt == ODP_PACKET_INVALID) return -1; @@ -1366,12 +1361,13 @@ int odp_packet_rem_data(odp_packet_t *pkt_ptr, uint32_t offset, uint32_t len) odp_packet_t pkt = *pkt_ptr; odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); uint32_t pktlen = pkt_hdr->frame_len; + pool_t *pool = pkt_hdr->buf_hdr.pool_ptr; odp_packet_t newpkt; if (offset > pktlen || offset + len > pktlen) return -1; - newpkt = odp_packet_alloc(pkt_hdr->buf_hdr.pool_hdl, pktlen - len); + newpkt = odp_packet_alloc(pool->pool_hdl, pktlen - len); if (newpkt == ODP_PACKET_INVALID) return -1; @@ -1436,12 +1432,12 @@ int odp_packet_concat(odp_packet_t *dst, odp_packet_t src) { odp_packet_hdr_t *dst_hdr = packet_hdr(*dst); odp_packet_hdr_t *src_hdr = packet_hdr(src); - int dst_segs = dst_hdr->buf_hdr.segcount; - int src_segs = src_hdr->buf_hdr.segcount; - odp_pool_t dst_pool = dst_hdr->buf_hdr.pool_hdl; - odp_pool_t src_pool = src_hdr->buf_hdr.pool_hdl; - uint32_t dst_len = dst_hdr->frame_len; - uint32_t src_len = src_hdr->frame_len; + int dst_segs = dst_hdr->buf_hdr.segcount; + int src_segs = src_hdr->buf_hdr.segcount; + pool_t *dst_pool = dst_hdr->buf_hdr.pool_ptr; + pool_t *src_pool = src_hdr->buf_hdr.pool_ptr; + uint32_t dst_len = dst_hdr->frame_len; + uint32_t src_len = src_hdr->frame_len; /* Do a copy if resulting packet would be out of segments or packets * are from different pools. */ @@ -1703,7 +1699,7 @@ void odp_packet_print(odp_packet_t pkt) int len = 0; int n = max_len - 1; odp_packet_hdr_t *hdr = packet_hdr(pkt); - odp_buffer_t buf = _odp_packet_to_buffer(pkt); + odp_buffer_t buf = packet_to_buffer(pkt); len += snprintf(&str[len], n - len, "Packet "); len += odp_buffer_snprint(&str[len], n - len, buf); @@ -1751,7 +1747,7 @@ void odp_packet_print(odp_packet_t pkt) int odp_packet_is_valid(odp_packet_t pkt) { - if (odp_buffer_is_valid(_odp_packet_to_buffer(pkt)) == 0) + if (odp_buffer_is_valid(packet_to_buffer(pkt)) == 0) return 0; if (odp_event_type(odp_packet_to_event(pkt)) != ODP_EVENT_PACKET) diff --git a/platform/linux-generic/odp_pool.c b/platform/linux-generic/odp_pool.c index 9dba7341..7e0c7606 100644 --- a/platform/linux-generic/odp_pool.c +++ b/platform/linux-generic/odp_pool.c @@ -52,12 +52,32 @@ static inline odp_pool_t pool_index_to_handle(uint32_t pool_idx) return _odp_cast_scalar(odp_pool_t, pool_idx); } -static inline uint32_t pool_id_from_buf(odp_buffer_t buf) +static inline pool_t *pool_from_buf(odp_buffer_t buf) { - odp_buffer_bits_t handle; + odp_buffer_hdr_t *buf_hdr = buf_hdl_to_hdr(buf); - handle.handle = buf; - return handle.pool_id; + return buf_hdr->pool_ptr; +} + +static inline odp_buffer_hdr_t *buf_hdr_from_index(pool_t *pool, + uint32_t buffer_idx) +{ + uint32_t block_offset; + odp_buffer_hdr_t *buf_hdr; + + block_offset = buffer_idx * pool->block_size; + + /* clang requires cast to uintptr_t */ + buf_hdr = (odp_buffer_hdr_t *)(uintptr_t)&pool->base_addr[block_offset]; + + return buf_hdr; +} + +static inline uint32_t buf_index_from_hdl(odp_buffer_t buf) +{ + odp_buffer_hdr_t *buf_hdr = buf_hdl_to_hdr(buf); + + return buf_hdr->index; } int odp_pool_init_global(void) @@ -141,16 +161,14 @@ static void flush_cache(pool_cache_t *cache, pool_t *pool) { ring_t *ring; uint32_t mask; - uint32_t cache_num, i, data; + uint32_t cache_num, i; ring = &pool->ring->hdr; mask = pool->ring_mask; cache_num = cache->num; - for (i = 0; i < cache_num; i++) { - data = (uint32_t)(uintptr_t)cache->buf[i]; - ring_enq(ring, mask, data); - } + for (i = 0; i < cache_num; i++) + ring_enq(ring, mask, cache->buf_index[i]); cache->num = 0; } @@ -202,23 +220,11 @@ static pool_t *reserve_pool(void) return NULL; } -static odp_buffer_t form_buffer_handle(uint32_t pool_idx, uint32_t buffer_idx) -{ - odp_buffer_bits_t bits; - - bits.handle = 0; - bits.pool_id = pool_idx; - bits.index = buffer_idx; - - return bits.handle; -} - static void init_buffers(pool_t *pool) { uint32_t i; odp_buffer_hdr_t *buf_hdr; odp_packet_hdr_t *pkt_hdr; - odp_buffer_t buf_hdl; void *addr; void *uarea = NULL; uint8_t *data; @@ -256,10 +262,12 @@ static void init_buffers(pool_t *pool) seg_size = pool->headroom + pool->data_size + pool->tailroom; /* Initialize buffer metadata */ + buf_hdr->index = i; buf_hdr->size = seg_size; buf_hdr->type = type; buf_hdr->event_type = type; buf_hdr->pool_hdl = pool->pool_hdl; + buf_hdr->pool_ptr = pool; buf_hdr->uarea_addr = uarea; /* Show user requested size through API */ buf_hdr->uarea_size = pool->params.pkt.uarea_size; @@ -275,11 +283,8 @@ static void init_buffers(pool_t *pool) buf_hdr->buf_end = &data[offset + pool->data_size + pool->tailroom]; - buf_hdl = form_buffer_handle(pool->pool_idx, i); - buf_hdr->handle.handle = buf_hdl; - - /* Store buffer into the global pool */ - ring_enq(ring, mask, (uint32_t)(uintptr_t)buf_hdl); + /* Store buffer index into the global pool */ + ring_enq(ring, mask, i); } } @@ -600,7 +605,7 @@ int odp_pool_info(odp_pool_t pool_hdl, odp_pool_info_t *info) } int buffer_alloc_multi(pool_t *pool, odp_buffer_t buf[], - odp_buffer_hdr_t *buf_hdr[], int max_num) + odp_buffer_hdr_t *buf_hdr_out[], int max_num) { ring_t *ring; uint32_t mask, i; @@ -626,10 +631,13 @@ int buffer_alloc_multi(pool_t *pool, odp_buffer_t buf[], /* Get buffers from the cache */ for (i = 0; i < num_ch; i++) { - buf[i] = cache->buf[cache_num - num_ch + i]; + uint32_t j = cache_num - num_ch + i; + + hdr = buf_hdr_from_index(pool, cache->buf_index[j]); + buf[i] = buf_from_buf_hdr(hdr); - if (odp_likely(buf_hdr != NULL)) - buf_hdr[i] = pool_buf_hdl_to_hdr(pool, buf[i]); + if (odp_likely(buf_hdr_out != NULL)) + buf_hdr_out[i] = hdr; } /* If needed, get more from the global pool */ @@ -651,18 +659,18 @@ int buffer_alloc_multi(pool_t *pool, odp_buffer_t buf[], for (i = 0; i < num_deq; i++) { uint32_t idx = num_ch + i; - buf[idx] = (odp_buffer_t)(uintptr_t)data[i]; - hdr = pool_buf_hdl_to_hdr(pool, buf[idx]); + hdr = buf_hdr_from_index(pool, data[i]); odp_prefetch(hdr); - if (odp_likely(buf_hdr != NULL)) - buf_hdr[idx] = hdr; + buf[idx] = buf_from_buf_hdr(hdr); + + if (odp_likely(buf_hdr_out != NULL)) + buf_hdr_out[idx] = hdr; } /* Cache extra buffers. Cache is currently empty. */ for (i = 0; i < cache_num; i++) - cache->buf[i] = (odp_buffer_t) - (uintptr_t)data[num_deq + i]; + cache->buf_index[i] = data[num_deq + i]; cache->num = cache_num; } else { @@ -672,26 +680,28 @@ int buffer_alloc_multi(pool_t *pool, odp_buffer_t buf[], return num_ch + num_deq; } -static inline void buffer_free_to_pool(uint32_t pool_id, +static inline void buffer_free_to_pool(pool_t *pool, const odp_buffer_t buf[], int num) { - pool_t *pool; int i; ring_t *ring; uint32_t mask; pool_cache_t *cache; uint32_t cache_num; - cache = local.cache[pool_id]; - pool = pool_entry(pool_id); + cache = local.cache[pool->pool_idx]; /* Special case of a very large free. Move directly to * the global pool. */ if (odp_unlikely(num > CONFIG_POOL_CACHE_SIZE)) { + uint32_t buf_index[num]; + ring = &pool->ring->hdr; mask = pool->ring_mask; for (i = 0; i < num; i++) - ring_enq(ring, mask, (uint32_t)(uintptr_t)buf[i]); + buf_index[i] = buf_index_from_hdl(buf[i]); + + ring_enq_multi(ring, mask, buf_index, num); return; } @@ -718,8 +728,7 @@ static inline void buffer_free_to_pool(uint32_t pool_id, index = cache_num - burst; for (i = 0; i < burst; i++) - data[i] = (uint32_t) - (uintptr_t)cache->buf[index + i]; + data[i] = cache->buf_index[index + i]; ring_enq_multi(ring, mask, data, burst); } @@ -728,14 +737,14 @@ static inline void buffer_free_to_pool(uint32_t pool_id, } for (i = 0; i < num; i++) - cache->buf[cache_num + i] = buf[i]; + cache->buf_index[cache_num + i] = buf_index_from_hdl(buf[i]); cache->num = cache_num + num; } void buffer_free_multi(const odp_buffer_t buf[], int num_total) { - uint32_t pool_id; + pool_t *pool; int num; int i; int first = 0; @@ -743,18 +752,19 @@ void buffer_free_multi(const odp_buffer_t buf[], int num_total) while (1) { num = 1; i = 1; - pool_id = pool_id_from_buf(buf[first]); + + pool = pool_from_buf(buf[first]); /* 'num' buffers are from the same pool */ if (num_total > 1) { for (i = first; i < num_total; i++) - if (pool_id != pool_id_from_buf(buf[i])) + if (pool != pool_from_buf(buf[i])) break; num = i - first; } - buffer_free_to_pool(pool_id, &buf[first], num); + buffer_free_to_pool(pool, &buf[first], num); if (i == num_total) return; @@ -871,9 +881,9 @@ void odp_pool_print(odp_pool_t pool_hdl) odp_pool_t odp_buffer_pool(odp_buffer_t buf) { - uint32_t pool_id = pool_id_from_buf(buf); + pool_t *pool = pool_from_buf(buf); - return pool_index_to_handle(pool_id); + return pool->pool_hdl; } void odp_pool_param_init(odp_pool_param_t *params) @@ -901,15 +911,15 @@ void seg_free_tail(odp_buffer_hdr_t *buf_hdr, int segcount) int odp_buffer_is_valid(odp_buffer_t buf) { - odp_buffer_bits_t handle; pool_t *pool; - handle.handle = buf; - - if (handle.pool_id >= ODP_CONFIG_POOLS) + if (buf == ODP_BUFFER_INVALID) return 0; - pool = pool_entry(handle.pool_id); + pool = pool_from_buf(buf); + + if (pool->pool_idx >= ODP_CONFIG_POOLS) + return 0; if (pool->reserved == 0) return 0; diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c index d52814b2..348d609a 100644 --- a/platform/linux-generic/odp_queue.c +++ b/platform/linux-generic/odp_queue.c @@ -559,7 +559,7 @@ static int queue_deq_multi(odp_queue_t handle, odp_event_t events[], int num) ret = queue->s.dequeue_multi(qentry_to_int(queue), buf_hdr, num); for (i = 0; i < ret; i++) - events[i] = odp_buffer_to_event(buf_hdr[i]->handle.handle); + events[i] = event_from_buf_hdr(buf_hdr[i]); return ret; } @@ -574,7 +574,7 @@ static odp_event_t queue_deq(odp_queue_t handle) buf_hdr = queue->s.dequeue(qentry_to_int(queue)); if (buf_hdr) - return odp_buffer_to_event(buf_hdr->handle.handle); + return event_from_buf_hdr(buf_hdr); return ODP_EVENT_INVALID; } @@ -686,7 +686,7 @@ int sched_cb_queue_deq_multi(uint32_t queue_index, odp_event_t ev[], int num) if (ret > 0) for (i = 0; i < ret; i++) - ev[i] = odp_buffer_to_event(buf_hdr[i]->handle.handle); + ev[i] = event_from_buf_hdr(buf_hdr[i]); return ret; } diff --git a/platform/linux-generic/odp_traffic_mngr.c b/platform/linux-generic/odp_traffic_mngr.c index ec2d331f..53e10471 100644 --- a/platform/linux-generic/odp_traffic_mngr.c +++ b/platform/linux-generic/odp_traffic_mngr.c @@ -104,7 +104,7 @@ static int queue_tm_reenq(queue_t queue, odp_buffer_hdr_t *buf_hdr) odp_tm_queue_t tm_queue = MAKE_ODP_TM_QUEUE((uint8_t *)queue - offsetof(tm_queue_obj_t, tm_qentry)); - odp_packet_t pkt = _odp_packet_from_buffer(buf_hdr->handle.handle); + odp_packet_t pkt = _odp_packet_from_buf_hdr(buf_hdr); return odp_tm_enq(tm_queue, pkt); } diff --git a/platform/linux-generic/pktio/ipc.c b/platform/linux-generic/pktio/ipc.c index f0facd8c..acdfc4ff 100644 --- a/platform/linux-generic/pktio/ipc.c +++ b/platform/linux-generic/pktio/ipc.c @@ -580,12 +580,13 @@ static int ipc_pktio_send_lockless(pktio_entry_t *pktio_entry, for (i = 0; i < len; i++) { odp_packet_t pkt = pkt_table[i]; pool_t *ipc_pool = pool_entry_from_hdl(pktio_entry->s.ipc.pool); - odp_buffer_bits_t handle; - uint32_t pkt_pool_id; + odp_packet_hdr_t *pkt_hdr; + pool_t *pool; - handle.handle = _odp_packet_to_buffer(pkt); - pkt_pool_id = handle.pool_id; - if (pkt_pool_id != ipc_pool->pool_idx) { + pkt_hdr = odp_packet_hdr(pkt); + pool = pkt_hdr->buf_hdr.pool_ptr; + + if (pool->pool_idx != ipc_pool->pool_idx) { odp_packet_t newpkt; newpkt = odp_packet_copy(pkt, pktio_entry->s.ipc.pool); ----------------------------------------------------------------------- Summary of changes: include/odp/arch/default/api/abi/buffer.h | 7 +- include/odp/arch/default/api/abi/event.h | 2 +- .../include/odp/api/plat/buffer_types.h | 6 +- .../include/odp/api/plat/event_types.h | 2 +- .../include/odp/api/plat/timer_types.h | 2 +- .../linux-generic/include/odp_buffer_inlines.h | 4 +- .../linux-generic/include/odp_buffer_internal.h | 23 ++-- .../linux-generic/include/odp_packet_internal.h | 5 +- platform/linux-generic/include/odp_pool_internal.h | 35 +----- platform/linux-generic/odp_buffer.c | 4 +- platform/linux-generic/odp_packet.c | 105 +++++++++--------- platform/linux-generic/odp_pool.c | 123 ++++++++++----------- platform/linux-generic/odp_queue.c | 74 ++++--------- platform/linux-generic/odp_traffic_mngr.c | 2 +- platform/linux-generic/pktio/ipc.c | 11 +- 15 files changed, 159 insertions(+), 246 deletions(-) hooks/post-receive --

9 years, 1 month

[COMMIT] branch master updated. v1.15.0.0-55-g7fc6d27e

by git＠git-us.linaro.org

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 7fc6d27e937b57b31360b07028388c811f8300dc (commit) via 104993f09d06c9ffc5ff1b2b57e44bfddefb3062 (commit) via c6dc5055a2acdf4f22e37823a86bb76b86f40b63 (commit) via 66898d97524683ae6f39b0624ce5a67f1cd8f603 (commit) via 51dc8377db6b67090224cab32a6663d7168c4e05 (commit) via 780073842dd6d2d52a52e9d59def2f13ae7d8427 (commit) via 89b7caa359acc5f213e233a3eb1ee1e9c661c246 (commit) via ce8c72bb9e8b1e162765b26b4249b7ecf1b32355 (commit) via a664b1d5e6e75a2b2110025c5bbf7ae727142bf4 (commit) from d9473fe65b531bfbe54d83d4e429d72e553c17dc (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 7fc6d27e937b57b31360b07028388c811f8300dc Author: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Date: Wed May 24 18:54:15 2017 +0300 travis: use pkg-config to discover ODP flags Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/.travis.yml b/.travis.yml index c0a08838..665d70f8 100644 --- a/.travis.yml +++ b/.travis.yml @@ -124,10 +124,10 @@ script: - echo "Checking linking and run from install..." - pushd $HOME - echo "Dynamic link.." - - ${CC} ${OLDPWD}/example/hello/odp_hello.c -o odp_hello_inst -I${HOME}/odp-install/include -L${HOME}/odp-install/lib -lodp-linux -L${OLDPWD}/dpdk/x86_64-native-linuxapp-gcc/lib -lrt -ldpdk -lpthread -lcrypto -lpcap -ldl + - ${CC} ${OLDPWD}/example/hello/odp_hello.c -o odp_hello_inst `PKG_CONFIG_PATH=${HOME}/odp-install/lib/pkgconfig pkg-config --cflags --libs libodp-linux` - LD_LIBRARY_PATH="${HOME}/odp-install/lib:$LD_LIBRARY_PATH" ./odp_hello_inst - echo "Static link.." - - ${CC} ${OLDPWD}/example/hello/odp_hello.c -o odp_hello_inst -I${HOME}/odp-install/include -L${HOME}/odp-install/lib -lodp-linux -L${OLDPWD}/dpdk/x86_64-native-linuxapp-gcc/lib -lrt -ldpdk -lpthread -lcrypto -lpcap -ldl -static + - ${CC} ${OLDPWD}/example/hello/odp_hello.c -o odp_hello_inst `PKG_CONFIG_PATH=${HOME}/odp-install/lib/pkgconfig pkg-config --cflags --libs libodp-linux --static` -static - ./odp_hello_inst jobs: commit 104993f09d06c9ffc5ff1b2b57e44bfddefb3062 Author: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Date: Wed May 31 14:09:27 2017 +0300 linux-gen: stop poisoning CPPFLAGS with Netmap flags Instead of poisoning global AM_CPPFLAGS with Netmap flags, use fine-grained controls for compiling with Netmap. Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/configure.ac b/configure.ac index d3059ffb..e4903c7a 100644 --- a/configure.ac +++ b/configure.ac @@ -209,7 +209,6 @@ AC_SUBST([testdir]) ########################################################################## # Set conditionals as computed within platform specific files ########################################################################## -AM_CONDITIONAL([netmap_support], [test x$netmap_support = xyes ]) AM_CONDITIONAL([SDK_INSTALL_PATH_], [test "x${SDK_INSTALL_PATH_}" = "x1"]) AM_CONDITIONAL([test_installdir], [test "$testdir" != ""]) AM_CONDITIONAL([cunit_support], [test x$cunit_support = xyes ]) diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index 5efd420a..26eba284 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -13,6 +13,7 @@ AM_CFLAGS += -D_ODP_PKTIO_IPC AM_CPPFLAGS += $(OPENSSL_CPPFLAGS) AM_CPPFLAGS += $(DPDK_CPPFLAGS) +AM_CPPFLAGS += $(NETMAP_CPPFLAGS) include_HEADERS = \ $(top_srcdir)/include/odp.h \ diff --git a/platform/linux-generic/m4/odp_netmap.m4 b/platform/linux-generic/m4/odp_netmap.m4 index 880e9d58..dea80626 100644 --- a/platform/linux-generic/m4/odp_netmap.m4 +++ b/platform/linux-generic/m4/odp_netmap.m4 @@ -15,14 +15,14 @@ AC_ARG_WITH([netmap-path], AC_HELP_STRING([--with-netmap-path=DIR path to netmap root directory], [(or in the default path if not specified).]), [NETMAP_PATH=$withval - AM_CPPFLAGS="$AM_CPPFLAGS -isystem $NETMAP_PATH/sys" + NETMAP_CPPFLAGS="-isystem $NETMAP_PATH/sys" netmap_support=yes],[]) ########################################################################## # Save and set temporary compilation flags ########################################################################## OLD_CPPFLAGS=$CPPFLAGS -CPPFLAGS="$AM_CPPFLAGS $CPPFLAGS" +CPPFLAGS="$NETMAP_CPPFLAGS $CPPFLAGS" ########################################################################## # Check for netmap availability @@ -32,6 +32,7 @@ then AC_CHECK_HEADERS([net/netmap_user.h], [], [AC_MSG_FAILURE(["can't find netmap header"])]) ODP_CFLAGS="$ODP_CFLAGS -DODP_NETMAP" + AC_SUBST([NETMAP_CPPFLAGS]) else netmap_support=no fi @@ -40,3 +41,5 @@ fi # Restore old saved variables ########################################################################## CPPFLAGS=$OLD_CPPFLAGS + +AM_CONDITIONAL([netmap_support], [test x$netmap_support = xyes ]) commit c6dc5055a2acdf4f22e37823a86bb76b86f40b63 Author: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Date: Thu Jun 1 13:46:03 2017 +0300 linux-gen: move timer_create check to platform directory Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/configure.ac b/configure.ac index 8e518a16..d3059ffb 100644 --- a/configure.ac +++ b/configure.ac @@ -363,8 +363,6 @@ AC_CONFIG_FILES([Makefile pkgconfig/libodphelper.pc ]) -AC_SEARCH_LIBS([timer_create],[rt posix4]) - ########################################################################## # distribute the changed variables among the Makefiles diff --git a/pkgconfig/libodp-linux.pc.in b/pkgconfig/libodp-linux.pc.in index 860c1cae..220dbc9c 100644 --- a/pkgconfig/libodp-linux.pc.in +++ b/pkgconfig/libodp-linux.pc.in @@ -7,5 +7,5 @@ Name: libodp-linux Description: The ODP packet processing engine Version: @PKGCONFIG_VERSION@ Libs: -L${libdir} -lodp-linux @DPDK_LIBS@ -Libs.private: @OPENSSL_STATIC_LIBS@ @DPDK_PMDS@ @DPDK_LIBS@ @PCAP_LIBS@ @PTHREAD_LIBS@ -lrt -lpthread @ATOMIC_LIBS@ +Libs.private: @OPENSSL_STATIC_LIBS@ @DPDK_PMDS@ @DPDK_LIBS@ @PCAP_LIBS@ @PTHREAD_LIBS@ @TIMER_LIBS@ -lpthread @ATOMIC_LIBS@ Cflags: -I${includedir} diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index 4bb166d3..5efd420a 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -261,6 +261,7 @@ __LIB__libodp_linux_la_LIBADD = $(ATOMIC_LIBS) __LIB__libodp_linux_la_LIBADD += $(OPENSSL_LIBS) __LIB__libodp_linux_la_LIBADD += $(DPDK_LIBS) $(DPDK_PMDS) __LIB__libodp_linux_la_LIBADD += $(PTHREAD_LIBS) +__LIB__libodp_linux_la_LIBADD += $(TIMER_LIBS) if HAVE_PCAP __LIB__libodp_linux_la_LIBADD += $(PCAP_LIBS) diff --git a/platform/linux-generic/m4/configure.m4 b/platform/linux-generic/m4/configure.m4 index c8164b8c..8e008de4 100644 --- a/platform/linux-generic/m4/configure.m4 +++ b/platform/linux-generic/m4/configure.m4 @@ -75,6 +75,7 @@ fi AC_SUBST([ATOMIC_LIBS]) m4_include([platform/linux-generic/m4/odp_pthread.m4]) +m4_include([platform/linux-generic/m4/odp_timer.m4]) m4_include([platform/linux-generic/m4/odp_openssl.m4]) m4_include([platform/linux-generic/m4/odp_pcap.m4]) m4_include([platform/linux-generic/m4/odp_netmap.m4]) diff --git a/platform/linux-generic/m4/odp_timer.m4 b/platform/linux-generic/m4/odp_timer.m4 new file mode 100644 index 00000000..3122c92b --- /dev/null +++ b/platform/linux-generic/m4/odp_timer.m4 @@ -0,0 +1,8 @@ +########################################################################## +# Check for POSIX timer functions +########################################################################## + +AC_CHECK_LIB([rt], [timer_create], [TIMER_LIBS="-lrt"], + [AC_CHECK_LIB([posix4], [timer_create], [TIMER_LIBS="-lposix4"], + [AC_MSG_FAILURE([timer_create not found])])]) +AC_SUBST([TIMER_LIBS]) commit 66898d97524683ae6f39b0624ce5a67f1cd8f603 Author: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Date: Wed May 31 14:09:27 2017 +0300 linux-gen: stop poisoning CFLAGS/LDFLAGS with threading flags Instead of poisoning global AM_CFLAGS/AM_LDFLAGS with threading flags, use fine-grained controls for compiling/linking with threading. Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/Makefile.inc b/platform/Makefile.inc index 1903d92d..738f0f9a 100644 --- a/platform/Makefile.inc +++ b/platform/Makefile.inc @@ -16,6 +16,8 @@ AM_CFLAGS += $(VISIBILITY_CFLAGS) #The implementation will need to retain the deprecated implementation AM_CFLAGS += -Wno-deprecated-declarations +AM_CFLAGS += @PTHREAD_CFLAGS@ + odpapispecincludedir= $(includedir)/odp/api/spec odpapispecinclude_HEADERS = \ $(top_srcdir)/include/odp/api/spec/align.h \ diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index 8497d1a2..4bb166d3 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -260,6 +260,7 @@ endif __LIB__libodp_linux_la_LIBADD = $(ATOMIC_LIBS) __LIB__libodp_linux_la_LIBADD += $(OPENSSL_LIBS) __LIB__libodp_linux_la_LIBADD += $(DPDK_LIBS) $(DPDK_PMDS) +__LIB__libodp_linux_la_LIBADD += $(PTHREAD_LIBS) if HAVE_PCAP __LIB__libodp_linux_la_LIBADD += $(PCAP_LIBS) diff --git a/platform/linux-generic/m4/odp_pthread.m4 b/platform/linux-generic/m4/odp_pthread.m4 index 7f391039..48615894 100644 --- a/platform/linux-generic/m4/odp_pthread.m4 +++ b/platform/linux-generic/m4/odp_pthread.m4 @@ -6,8 +6,3 @@ AX_PTHREAD([CC="$PTHREAD_CC"], [ echo "Error! We require pthreads to be available" exit -1 ]) -LIBS="$PTHREAD_LIBS $LIBS" -AM_CFLAGS="$AM_CFLAGS $PTHREAD_CFLAGS" -AM_LDFLAGS="$AM_LDFLAGS $PTHREAD_LDFLAGS" - -AM_LDFLAGS="$AM_LDFLAGS -pthread -lrt" commit 51dc8377db6b67090224cab32a6663d7168c4e05 Author: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Date: Wed May 31 14:09:27 2017 +0300 linux-gen: stop poisoning CPPFLAGS/LDFLAGS with CUnit flags Instead of poisoning global AM_CPPFLAGS/AM_LDFLAGS with CUnit flags, use fine-grained controls for compiling/linking with CUnit. Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/test/common_plat/common/Makefile.am b/test/common_plat/common/Makefile.am index fd41fb42..7d88d2ea 100644 --- a/test/common_plat/common/Makefile.am +++ b/test/common_plat/common/Makefile.am @@ -4,6 +4,7 @@ include $(top_srcdir)/test/Makefile.inc noinst_LTLIBRARIES = libcunit_common.la libcpumask_common.la libthrmask_common.la libcunit_common_la_SOURCES = odp_cunit_common.c +libcunit_common_la_LIBADD = $(CUNIT_LIBS) libcpumask_common_la_SOURCES = mask_common.c diff --git a/test/common_plat/m4/validation.m4 b/test/common_plat/m4/validation.m4 index d32f675a..67edac7a 100644 --- a/test/common_plat/m4/validation.m4 +++ b/test/common_plat/m4/validation.m4 @@ -26,24 +26,24 @@ AC_ARG_WITH([cunit-path], AC_HELP_STRING([--with-cunit-path=DIR path to CUnit libs and headers], [(or in the default path if not specified).]), [CUNIT_PATH=$withval - AM_CPPFLAGS="$AM_CPPFLAGS -I$CUNIT_PATH/include" - AM_LDFLAGS="$AM_LDFLAGS -L$CUNIT_PATH/lib" - cunit_support=yes],[]) + CUNIT_CPPFLAGS="-I$CUNIT_PATH/include" + CUNIT_LIBS="-L$CUNIT_PATH/lib" + cunit_support=yes],[]) ########################################################################## # Save and set temporary compilation flags ########################################################################## -OLD_LDFLAGS=$LDFLAGS +OLD_LIBS=$LIBS OLD_CPPFLAGS=$CPPFLAGS -LDFLAGS="$AM_LDFLAGS $LDFLAGS" -CPPFLAGS="$AM_CPPFLAGS $CPPFLAGS" +LIBS="$CUNIT_LIBS $LIBS" +CPPFLAGS="$CUNIT_CPPFLAGS $CPPFLAGS" ########################################################################## # Check for CUnit availability ########################################################################## if test x$cunit_support = xyes then - AC_CHECK_LIB([cunit],[CU_get_error], [], + AC_CHECK_LIB([cunit],[CU_get_error], [CUNIT_LIBS="$CUNIT_LIBS -lcunit"], [AC_MSG_ERROR([CUnit libraries required])]) AC_CHECK_HEADERS([CUnit/Basic.h], [], [AC_MSG_FAILURE(["can't find cunit headers"])]) @@ -51,8 +51,11 @@ else cunit_support=no fi +AC_SUBST([CUNIT_CPPFLAGS]) +AC_SUBST([CUNIT_LIBS]) + ########################################################################## # Restore old saved variables ########################################################################## -LDFLAGS=$OLD_LDFLAGS +LIBS=$OLD_LIBS CPPFLAGS=$OLD_CPPFLAGS diff --git a/test/common_plat/validation/api/Makefile.inc b/test/common_plat/validation/api/Makefile.inc index 6069ee6b..177d2ac3 100644 --- a/test/common_plat/validation/api/Makefile.inc +++ b/test/common_plat/validation/api/Makefile.inc @@ -11,6 +11,8 @@ AM_CFLAGS += -I$(top_srcdir)/test/common_plat/common AM_LDFLAGS += -static AM_LDFLAGS += $(DPDK_PMDS) +AM_CPPFLAGS += $(CUNIT_CPPFLAGS) + LIBCUNIT_COMMON = $(COMMON_DIR)/libcunit_common.la LIBCPUMASK_COMMON = $(COMMON_DIR)/libcpumask_common.la LIBTHRMASK_COMMON = $(COMMON_DIR)/libthrmask_common.la diff --git a/test/linux-generic/Makefile.inc b/test/linux-generic/Makefile.inc index 6e165d8d..30b56e26 100644 --- a/test/linux-generic/Makefile.inc +++ b/test/linux-generic/Makefile.inc @@ -4,6 +4,8 @@ AM_LDFLAGS += -static +AM_CPPFLAGS += $(CUNIT_CPPFLAGS) + LIBCUNIT_COMMON = $(top_builddir)/test/common_plat/common/libcunit_common.la LIB = $(top_builddir)/lib LIBODP = $(LIB)/libodphelper.la $(LIB)/libodp-linux.la $(DPDK_PMDS) commit 780073842dd6d2d52a52e9d59def2f13ae7d8427 Author: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Date: Wed May 31 14:09:27 2017 +0300 linux-gen: stop poisoning CFLAGS/LIBS with libpcap flags Instead of poisoning global AM_CFLAGS/LIBS with libpcap flags, use fine-grained controls for compiling/linking with libpcap. Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/configure.ac b/configure.ac index 3432b29a..8e518a16 100644 --- a/configure.ac +++ b/configure.ac @@ -210,7 +210,6 @@ AC_SUBST([testdir]) # Set conditionals as computed within platform specific files ########################################################################## AM_CONDITIONAL([netmap_support], [test x$netmap_support = xyes ]) -AM_CONDITIONAL([HAVE_PCAP], [test $have_pcap = yes]) AM_CONDITIONAL([SDK_INSTALL_PATH_], [test "x${SDK_INSTALL_PATH_}" = "x1"]) AM_CONDITIONAL([test_installdir], [test "$testdir" != ""]) AM_CONDITIONAL([cunit_support], [test x$cunit_support = xyes ]) diff --git a/pkgconfig/libodp-linux.pc.in b/pkgconfig/libodp-linux.pc.in index 6dc06dc9..860c1cae 100644 --- a/pkgconfig/libodp-linux.pc.in +++ b/pkgconfig/libodp-linux.pc.in @@ -7,5 +7,5 @@ Name: libodp-linux Description: The ODP packet processing engine Version: @PKGCONFIG_VERSION@ Libs: -L${libdir} -lodp-linux @DPDK_LIBS@ -Libs.private: @OPENSSL_STATIC_LIBS@ @DPDK_PMDS@ @DPDK_LIBS@ -lpcap @PTHREAD_LIBS@ -lrt -lpthread @ATOMIC_LIBS@ +Libs.private: @OPENSSL_STATIC_LIBS@ @DPDK_PMDS@ @DPDK_LIBS@ @PCAP_LIBS@ @PTHREAD_LIBS@ -lrt -lpthread @ATOMIC_LIBS@ Cflags: -I${includedir} diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index 2215e575..8497d1a2 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -261,6 +261,10 @@ __LIB__libodp_linux_la_LIBADD = $(ATOMIC_LIBS) __LIB__libodp_linux_la_LIBADD += $(OPENSSL_LIBS) __LIB__libodp_linux_la_LIBADD += $(DPDK_LIBS) $(DPDK_PMDS) +if HAVE_PCAP +__LIB__libodp_linux_la_LIBADD += $(PCAP_LIBS) +endif + # Create symlink for ABI header files. Application does not need to use the arch # specific include path for installed files. install-data-hook: diff --git a/platform/linux-generic/m4/odp_pcap.m4 b/platform/linux-generic/m4/odp_pcap.m4 index 0439c60b..ffdff17c 100644 --- a/platform/linux-generic/m4/odp_pcap.m4 +++ b/platform/linux-generic/m4/odp_pcap.m4 @@ -9,6 +9,10 @@ AC_CHECK_HEADER(pcap/pcap.h, []) if test $have_pcap == yes; then - AM_CFLAGS="$AM_CFLAGS -DHAVE_PCAP" - LIBS="$LIBS -lpcap" + ODP_CFLAGS="$AM_CFLAGS -DHAVE_PCAP" + PCAP_LIBS="-lpcap" fi + +AC_SUBST([PCAP_LIBS]) + +AM_CONDITIONAL([HAVE_PCAP], [test $have_pcap = yes]) commit 89b7caa359acc5f213e233a3eb1ee1e9c661c246 Author: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Date: Wed May 31 14:09:27 2017 +0300 linux-gen: stop poisoning CPPFLAGS/LDFLAGS with DPDK flags Instead of poisoning global AM_CPPFLAGS/AM_LDFLAGS with DPDK flags, use fine-grained controls for compiling/linking with DPDK. Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/configure.ac b/configure.ac index 45812f66..3432b29a 100644 --- a/configure.ac +++ b/configure.ac @@ -210,7 +210,6 @@ AC_SUBST([testdir]) # Set conditionals as computed within platform specific files ########################################################################## AM_CONDITIONAL([netmap_support], [test x$netmap_support = xyes ]) -AM_CONDITIONAL([PKTIO_DPDK], [test x$pktio_dpdk_support = xyes ]) AM_CONDITIONAL([HAVE_PCAP], [test $have_pcap = yes]) AM_CONDITIONAL([SDK_INSTALL_PATH_], [test "x${SDK_INSTALL_PATH_}" = "x1"]) AM_CONDITIONAL([test_installdir], [test "$testdir" != ""]) diff --git a/pkgconfig/libodp-linux.pc.in b/pkgconfig/libodp-linux.pc.in index 0c5883b6..6dc06dc9 100644 --- a/pkgconfig/libodp-linux.pc.in +++ b/pkgconfig/libodp-linux.pc.in @@ -6,6 +6,6 @@ includedir=@includedir@ Name: libodp-linux Description: The ODP packet processing engine Version: @PKGCONFIG_VERSION@ -Libs: -L${libdir} -lodp-linux -Libs.private: @OPENSSL_STATIC_LIBS@ -lpcap @PTHREAD_LIBS@ -lrt -lpthread @ATOMIC_LIBS@ +Libs: -L${libdir} -lodp-linux @DPDK_LIBS@ +Libs.private: @OPENSSL_STATIC_LIBS@ @DPDK_PMDS@ @DPDK_LIBS@ -lpcap @PTHREAD_LIBS@ -lrt -lpthread @ATOMIC_LIBS@ Cflags: -I${includedir} diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index feaa94ab..2215e575 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -12,6 +12,7 @@ AM_CFLAGS += -Iinclude AM_CFLAGS += -D_ODP_PKTIO_IPC AM_CPPFLAGS += $(OPENSSL_CPPFLAGS) +AM_CPPFLAGS += $(DPDK_CPPFLAGS) include_HEADERS = \ $(top_srcdir)/include/odp.h \ @@ -258,6 +259,7 @@ endif __LIB__libodp_linux_la_LIBADD = $(ATOMIC_LIBS) __LIB__libodp_linux_la_LIBADD += $(OPENSSL_LIBS) +__LIB__libodp_linux_la_LIBADD += $(DPDK_LIBS) $(DPDK_PMDS) # Create symlink for ABI header files. Application does not need to use the arch # specific include path for installed files. diff --git a/platform/linux-generic/m4/odp_dpdk.m4 b/platform/linux-generic/m4/odp_dpdk.m4 index 58d14727..1995e0fe 100644 --- a/platform/linux-generic/m4/odp_dpdk.m4 +++ b/platform/linux-generic/m4/odp_dpdk.m4 @@ -4,15 +4,15 @@ pktio_dpdk_support=no AC_ARG_WITH([dpdk-path], AC_HELP_STRING([--with-dpdk-path=DIR path to dpdk build directory]), - [DPDK_PATH=$withval - AM_CPPFLAGS="$AM_CPPFLAGS -msse4.2 -isystem $DPDK_PATH/include" + [DPDK_PATH="$withval" + DPDK_CPPFLAGS="-msse4.2 -isystem $DPDK_PATH/include" pktio_dpdk_support=yes],[]) ########################################################################## # Save and set temporary compilation flags ########################################################################## -OLD_CPPFLAGS=$CPPFLAGS -CPPFLAGS="$AM_CPPFLAGS $CPPFLAGS" +OLD_CPPFLAGS="$CPPFLAGS" +CPPFLAGS="$DPDK_CPPFLAGS $CPPFLAGS" ########################################################################## # Check for DPDK availability @@ -25,22 +25,23 @@ then AC_CHECK_HEADERS([rte_config.h], [], [AC_MSG_FAILURE(["can't find DPDK header"])]) - DPDK_PMD=--whole-archive, - for filename in $with_dpdk_path/lib/*.a; do - cur_driver=`echo $(basename "$filename" .a) | \ - sed -n 's/^$librte_pmd_$/-lrte_pmd_/p' | sed -n 's/$/,/p'` + AS_VAR_SET([DPDK_PMDS], [-Wl,--whole-archive,]) + for filename in "$DPDK_PATH"/lib/librte_pmd_*.a; do + cur_driver=`basename "$filename" .a | sed -e 's/^lib//'` # rte_pmd_nfp has external dependencies which break linking - if test "$cur_driver" = "-lrte_pmd_nfp,"; then + if test "$cur_driver" = "rte_pmd_nfp"; then echo "skip linking rte_pmd_nfp" else - DPDK_PMD+=$cur_driver + AS_VAR_APPEND([DPDK_PMDS], [-l$cur_driver,]) fi done - DPDK_PMD+=--no-whole-archive + AS_VAR_APPEND([DPDK_PMDS], [--no-whole-archive]) ODP_CFLAGS="$ODP_CFLAGS -DODP_PKTIO_DPDK" - AM_LDFLAGS="$AM_LDFLAGS -L$DPDK_PATH/lib -Wl,$DPDK_PMD" - LIBS="$LIBS -ldpdk -ldl -lpcap" + DPDK_LIBS="-L$DPDK_PATH/lib -ldpdk -lpthread -ldl -lpcap" + AC_SUBST([DPDK_CPPFLAGS]) + AC_SUBST([DPDK_LIBS]) + AC_SUBST([DPDK_PMDS]) else pktio_dpdk_support=no fi @@ -49,3 +50,5 @@ fi # Restore old saved variables ########################################################################## CPPFLAGS=$OLD_CPPFLAGS + +AM_CONDITIONAL([PKTIO_DPDK], [test x$pktio_dpdk_support = xyes ]) diff --git a/test/Makefile.inc b/test/Makefile.inc index 1ef2a92c..bf31b374 100644 --- a/test/Makefile.inc +++ b/test/Makefile.inc @@ -4,7 +4,7 @@ LIB = $(top_builddir)/lib #in the following line, the libs using the symbols should come before #the libs containing them! The includer is given a chance to add things #before libodp by setting PRE_LDADD before the inclusion. -LDADD = $(PRE_LDADD) $(LIB)/libodphelper.la $(LIB)/libodp-linux.la +LDADD = $(PRE_LDADD) $(LIB)/libodphelper.la $(LIB)/libodp-linux.la $(DPDK_PMDS) INCFLAGS = \ -I$(top_builddir)/platform/@with_platform@/include \ diff --git a/test/common_plat/validation/api/Makefile.inc b/test/common_plat/validation/api/Makefile.inc index dda18adc..6069ee6b 100644 --- a/test/common_plat/validation/api/Makefile.inc +++ b/test/common_plat/validation/api/Makefile.inc @@ -9,6 +9,7 @@ AUTOMAKE_OPTIONS = nostdinc AM_CFLAGS += -I$(top_srcdir)/test/common_plat/common AM_LDFLAGS += -static +AM_LDFLAGS += $(DPDK_PMDS) LIBCUNIT_COMMON = $(COMMON_DIR)/libcunit_common.la LIBCPUMASK_COMMON = $(COMMON_DIR)/libcpumask_common.la diff --git a/test/linux-generic/Makefile.inc b/test/linux-generic/Makefile.inc index 198087f3..6e165d8d 100644 --- a/test/linux-generic/Makefile.inc +++ b/test/linux-generic/Makefile.inc @@ -6,7 +6,7 @@ AM_LDFLAGS += -static LIBCUNIT_COMMON = $(top_builddir)/test/common_plat/common/libcunit_common.la LIB = $(top_builddir)/lib -LIBODP = $(LIB)/libodphelper.la $(LIB)/libodp-linux.la +LIBODP = $(LIB)/libodphelper.la $(LIB)/libodp-linux.la $(DPDK_PMDS) INCCUNIT_COMMON = -I$(top_srcdir)/test/common_plat/common INCODP = \ commit ce8c72bb9e8b1e162765b26b4249b7ecf1b32355 Author: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Date: Wed May 31 14:09:27 2017 +0300 linux-gen: stop poisoning CPPFLAGS/LDFLAGS with OpenSSL flags Instead of poisoning global AM_CPPFLAGS/AM_LDFLAGS with OpenSSL flags, use fine-grained controls for compiling/linking with OpenSSL. Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/pkgconfig/libodp-linux.pc.in b/pkgconfig/libodp-linux.pc.in index 1a8d489f..0c5883b6 100644 --- a/pkgconfig/libodp-linux.pc.in +++ b/pkgconfig/libodp-linux.pc.in @@ -7,5 +7,5 @@ Name: libodp-linux Description: The ODP packet processing engine Version: @PKGCONFIG_VERSION@ Libs: -L${libdir} -lodp-linux -Libs.private: -lcrypto -ldl -lpcap @PTHREAD_LIBS@ -lrt -lpthread @ATOMIC_LIBS@ +Libs.private: @OPENSSL_STATIC_LIBS@ -lpcap @PTHREAD_LIBS@ -lrt -lpthread @ATOMIC_LIBS@ Cflags: -I${includedir} diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index 33299564..feaa94ab 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -11,6 +11,8 @@ AM_CFLAGS += -I$(top_builddir)/include AM_CFLAGS += -Iinclude AM_CFLAGS += -D_ODP_PKTIO_IPC +AM_CPPFLAGS += $(OPENSSL_CPPFLAGS) + include_HEADERS = \ $(top_srcdir)/include/odp.h \ $(top_srcdir)/include/odp_api.h @@ -255,6 +257,7 @@ __LIB__libodp_linux_la_SOURCES += pktio/pcap.c endif __LIB__libodp_linux_la_LIBADD = $(ATOMIC_LIBS) +__LIB__libodp_linux_la_LIBADD += $(OPENSSL_LIBS) # Create symlink for ABI header files. Application does not need to use the arch # specific include path for installed files. diff --git a/platform/linux-generic/m4/odp_openssl.m4 b/platform/linux-generic/m4/odp_openssl.m4 index 2344914b..1d520077 100644 --- a/platform/linux-generic/m4/odp_openssl.m4 +++ b/platform/linux-generic/m4/odp_openssl.m4 @@ -5,8 +5,8 @@ AC_ARG_WITH([openssl-path], AC_HELP_STRING([--with-openssl-path=DIR path to openssl libs and headers], [(or in the default path if not specified).]), [OPENSSL_PATH=$withval - AM_CPPFLAGS="$AM_CPPFLAGS -I$OPENSSL_PATH/include" - AM_LDFLAGS="$AM_LDFLAGS -L$OPENSSL_PATH/lib" + OPENSSL_CPPFLAGS="-I$OPENSSL_PATH/include" + OPENSSL_LIBS="-L$OPENSSL_PATH/lib" ],[]) ########################################################################## @@ -14,19 +14,24 @@ AC_HELP_STRING([--with-openssl-path=DIR path to openssl libs and headers], ########################################################################## OLD_LDFLAGS=$LDFLAGS OLD_CPPFLAGS=$CPPFLAGS -LDFLAGS="$AM_LDFLAGS $LDFLAGS" -CPPFLAGS="$AM_CPPFLAGS $CPPFLAGS" +LIBS="$OPENSSL_LIBS $LIBS" +CPPFLAGS="$OPENSSL_CPPFLAGS $CPPFLAGS" ########################################################################## # Check for OpenSSL availability ########################################################################## -AC_CHECK_LIB([crypto], [EVP_EncryptInit], [], +AC_CHECK_LIB([crypto], [EVP_EncryptInit], [OPENSSL_LIBS="$OPENSSL_LIBS -lcrypto" + OPENSSL_STATIC_LIBS="$OPENSSL_LIBS -ldl"], [AC_MSG_FAILURE([OpenSSL libraries required])]) AC_CHECK_HEADERS([openssl/des.h openssl/rand.h openssl/hmac.h openssl/evp.h], [], [AC_MSG_ERROR([OpenSSL headers required])]) +AC_SUBST([OPENSSL_CPPFLAGS]) +AC_SUBST([OPENSSL_LIBS]) +AC_SUBST([OPENSSL_STATIC_LIBS]) + ########################################################################## # Restore old saved variables ########################################################################## -LDFLAGS=$OLD_LDFLAGS +LIBS=$OLD_LIBS CPPFLAGS=$OLD_CPPFLAGS commit a664b1d5e6e75a2b2110025c5bbf7ae727142bf4 Author: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Date: Wed May 31 14:08:03 2017 +0300 pkgconfig: provide minimal proper static linking flags Provide Libs.private for static linking with ODP in minimal case (no DPDK, no netmap). Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/pkgconfig/libodp-linux.pc.in b/pkgconfig/libodp-linux.pc.in index 61770175..1a8d489f 100644 --- a/pkgconfig/libodp-linux.pc.in +++ b/pkgconfig/libodp-linux.pc.in @@ -7,5 +7,5 @@ Name: libodp-linux Description: The ODP packet processing engine Version: @PKGCONFIG_VERSION@ Libs: -L${libdir} -lodp-linux -Libs.private: @ATOMIC_LIBS@ +Libs.private: -lcrypto -ldl -lpcap @PTHREAD_LIBS@ -lrt -lpthread @ATOMIC_LIBS@ Cflags: -I${includedir} ----------------------------------------------------------------------- Summary of changes: .travis.yml | 4 ++-- configure.ac | 5 ----- pkgconfig/libodp-linux.pc.in | 4 ++-- platform/Makefile.inc | 2 ++ platform/linux-generic/Makefile.am | 12 ++++++++++++ platform/linux-generic/m4/configure.m4 | 1 + platform/linux-generic/m4/odp_dpdk.m4 | 29 +++++++++++++++------------- platform/linux-generic/m4/odp_netmap.m4 | 7 +++++-- platform/linux-generic/m4/odp_openssl.m4 | 17 ++++++++++------ platform/linux-generic/m4/odp_pcap.m4 | 8 ++++++-- platform/linux-generic/m4/odp_pthread.m4 | 5 ----- platform/linux-generic/m4/odp_timer.m4 | 8 ++++++++ test/Makefile.inc | 2 +- test/common_plat/common/Makefile.am | 1 + test/common_plat/m4/validation.m4 | 19 ++++++++++-------- test/common_plat/validation/api/Makefile.inc | 3 +++ test/linux-generic/Makefile.inc | 4 +++- 17 files changed, 84 insertions(+), 47 deletions(-) create mode 100644 platform/linux-generic/m4/odp_timer.m4 hooks/post-receive --

9 years, 1 month

[COMMIT] branch api-next updated. v1.15.0.0-419-gee5be324

by git＠git-us.linaro.org

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, api-next has been updated via ee5be324411a7520528a367967c28fc529d3bc2e (commit) via 533a2fe155aa16268bfe4091a3a056a78d5dbfc6 (commit) via b4edd15ea49b38a0c1ed2bd021ee36aa296a79b0 (commit) via 99b74cbb3c77902322933192ce68f8c1abe21e84 (commit) via d0e3acc0cfdab0d4c66b49ea70207d8938569b80 (commit) via c6e3edd5dc52764e4f1772341dc568a549d7eaa9 (commit) via 26644862f90f4dc2f6ed99534d16f8b4d136de82 (commit) via e777674f2d484e98e3329c3e22022fb4c6d86069 (commit) from 0275cc02d5cdb54b955cfce1988f5ee158ad0423 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit ee5be324411a7520528a367967c28fc529d3bc2e Author: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Date: Sat Jul 1 00:34:54 2017 +0300 example: ipsec: rewrite using Crypto packet API Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Petri Savolainen <petri.savolainen(a)nokia.com> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/example/ipsec/odp_ipsec.c b/example/ipsec/odp_ipsec.c index c618cc46..5df8f154 100644 --- a/example/ipsec/odp_ipsec.c +++ b/example/ipsec/odp_ipsec.c @@ -144,7 +144,7 @@ typedef struct { uint32_t dst_ip; /**< SA dest IP address */ /* Output only */ - odp_crypto_op_param_t params; /**< Parameters for crypto call */ + odp_crypto_packet_op_param_t params; /**< Parameters for crypto call */ uint32_t *ah_seq; /**< AH sequence number location */ uint32_t *esp_seq; /**< ESP sequence number location */ uint16_t *tun_hdr_id; /**< Tunnel header ID > */ @@ -393,7 +393,9 @@ void ipsec_init_post(crypto_api_mode_e api_mode) auth_sa, tun, api_mode, - entry->input)) { + entry->input, + completionq, + out_pool)) { EXAMPLE_ERR("Error: IPSec cache entry failed.\n" ); exit(EXIT_FAILURE); @@ -627,19 +629,18 @@ pkt_disposition_e do_route_fwd_db(odp_packet_t pkt, pkt_ctx_t *ctx) * @return PKT_CONTINUE if done else PKT_POSTED */ static -pkt_disposition_e do_ipsec_in_classify(odp_packet_t pkt, +pkt_disposition_e do_ipsec_in_classify(odp_packet_t *pkt, pkt_ctx_t *ctx, - odp_bool_t *skip, - odp_crypto_op_result_t *result) + odp_bool_t *skip) { - uint8_t *buf = odp_packet_data(pkt); - odph_ipv4hdr_t *ip = (odph_ipv4hdr_t *)odp_packet_l3_ptr(pkt, NULL); + uint8_t *buf = odp_packet_data(*pkt); + odph_ipv4hdr_t *ip = (odph_ipv4hdr_t *)odp_packet_l3_ptr(*pkt, NULL); int hdr_len; odph_ahhdr_t *ah = NULL; odph_esphdr_t *esp = NULL; ipsec_cache_entry_t *entry; - odp_crypto_op_param_t params; - odp_bool_t posted = 0; + odp_crypto_packet_op_param_t params; + odp_packet_t out_pkt; /* Default to skip IPsec */ *skip = TRUE; @@ -661,8 +662,7 @@ pkt_disposition_e do_ipsec_in_classify(odp_packet_t pkt, /* Initialize parameters block */ memset(&params, 0, sizeof(params)); params.session = entry->state.session; - params.pkt = pkt; - params.out_pkt = entry->in_place ? pkt : ODP_PACKET_INVALID; + out_pkt = entry->in_place ? *pkt : ODP_PACKET_INVALID; /*Save everything to context */ ctx->ipsec.ip_tos = ip->tos; @@ -697,12 +697,17 @@ pkt_disposition_e do_ipsec_in_classify(odp_packet_t pkt, /* Issue crypto request */ *skip = FALSE; ctx->state = PKT_STATE_IPSEC_IN_FINISH; - if (odp_crypto_operation(&params, - &posted, - result)) { - abort(); + if (entry->async) { + if (odp_crypto_packet_op_enq(pkt, &out_pkt, &params, 1)) + abort(); + return PKT_POSTED; } - return (posted) ? PKT_POSTED : PKT_CONTINUE; + + if (odp_crypto_packet_op(pkt, &out_pkt, &params, 1)) + abort(); + *pkt = out_pkt; + + return PKT_CONTINUE; } /** @@ -715,18 +720,20 @@ pkt_disposition_e do_ipsec_in_classify(odp_packet_t pkt, */ static pkt_disposition_e do_ipsec_in_finish(odp_packet_t pkt, - pkt_ctx_t *ctx, - odp_crypto_op_result_t *result) + pkt_ctx_t *ctx) { odph_ipv4hdr_t *ip; + odp_crypto_packet_result_t result; int hdr_len = ctx->ipsec.hdr_len; int trl_len = 0; + odp_crypto_packet_result(&result, pkt); + /* Check crypto result */ - if (!result->ok) { - if (!is_crypto_op_status_ok(&result->cipher_status)) + if (!result.ok) { + if (!is_crypto_op_status_ok(&result.cipher_status)) return PKT_DROP; - if (!is_crypto_op_status_ok(&result->auth_status)) + if (!is_crypto_op_status_ok(&result.auth_status)) return PKT_DROP; } ip = (odph_ipv4hdr_t *)odp_packet_l3_ptr(pkt, NULL); @@ -816,7 +823,7 @@ pkt_disposition_e do_ipsec_out_classify(odp_packet_t pkt, uint16_t ip_data_len = ipv4_data_len(ip); uint8_t *ip_data = ipv4_data_p(ip); ipsec_cache_entry_t *entry; - odp_crypto_op_param_t params; + odp_crypto_packet_op_param_t params; int hdr_len = 0; int trl_len = 0; odph_ahhdr_t *ah = NULL; @@ -840,8 +847,6 @@ pkt_disposition_e do_ipsec_out_classify(odp_packet_t pkt, /* Initialize parameters block */ memset(&params, 0, sizeof(params)); params.session = entry->state.session; - params.pkt = pkt; - params.out_pkt = entry->in_place ? pkt : ODP_PACKET_INVALID; if (entry->mode == IPSEC_SA_MODE_TUNNEL) { hdr_len += sizeof(odph_ipv4hdr_t); @@ -949,12 +954,19 @@ pkt_disposition_e do_ipsec_out_classify(odp_packet_t pkt, * @return PKT_CONTINUE if done else PKT_POSTED */ static -pkt_disposition_e do_ipsec_out_seq(odp_packet_t pkt, - pkt_ctx_t *ctx, - odp_crypto_op_result_t *result) +pkt_disposition_e do_ipsec_out_seq(odp_packet_t *pkt, + pkt_ctx_t *ctx) { - uint8_t *buf = odp_packet_data(pkt); - odp_bool_t posted = 0; + uint8_t *buf = odp_packet_data(*pkt); + odph_ipv4hdr_t *ip = (odph_ipv4hdr_t *)odp_packet_l3_ptr(*pkt, NULL); + odp_packet_t out_pkt; + ipsec_cache_entry_t *entry; + + entry = find_ipsec_cache_entry_out(odp_be_to_cpu_32(ip->src_addr), + odp_be_to_cpu_32(ip->dst_addr), + ip->proto); + if (!entry) + return PKT_DROP; /* We were dispatched from atomic queue, assign sequence numbers */ if (ctx->ipsec.ah_offset) { @@ -985,13 +997,22 @@ pkt_disposition_e do_ipsec_out_seq(odp_packet_t pkt, } } + out_pkt = entry->in_place ? *pkt : ODP_PACKET_INVALID; + /* Issue crypto request */ - if (odp_crypto_operation(&ctx->ipsec.params, - &posted, - result)) { - abort(); + if (entry->async) { + if (odp_crypto_packet_op_enq(pkt, &out_pkt, + &ctx->ipsec.params, 1)) + abort(); + return PKT_POSTED; } - return (posted) ? PKT_POSTED : PKT_CONTINUE; + + if (odp_crypto_packet_op(pkt, &out_pkt, + &ctx->ipsec.params, 1)) + abort(); + *pkt = out_pkt; + + return PKT_CONTINUE; } /** @@ -1004,16 +1025,18 @@ pkt_disposition_e do_ipsec_out_seq(odp_packet_t pkt, */ static pkt_disposition_e do_ipsec_out_finish(odp_packet_t pkt, - pkt_ctx_t *ctx, - odp_crypto_op_result_t *result) + pkt_ctx_t *ctx) { odph_ipv4hdr_t *ip; + odp_crypto_packet_result_t result; + + odp_crypto_packet_result(&result, pkt); /* Check crypto result */ - if (!result->ok) { - if (!is_crypto_op_status_ok(&result->cipher_status)) + if (!result.ok) { + if (!is_crypto_op_status_ok(&result.cipher_status)) return PKT_DROP; - if (!is_crypto_op_status_ok(&result->auth_status)) + if (!is_crypto_op_status_ok(&result.auth_status)) return PKT_DROP; } ip = (odph_ipv4hdr_t *)odp_packet_l3_ptr(pkt, NULL); @@ -1063,15 +1086,15 @@ int pktio_thread(void *arg EXAMPLE_UNUSED) pkt_disposition_e rc; pkt_ctx_t *ctx; odp_queue_t dispatchq; - odp_crypto_op_result_t result; + odp_event_subtype_t subtype; /* Use schedule to get event from any input queue */ ev = schedule(&dispatchq); /* Determine new work versus completion or sequence number */ - if (ODP_EVENT_PACKET == odp_event_type(ev)) { + if (ODP_EVENT_PACKET == odp_event_types(ev, &subtype)) { pkt = odp_packet_from_event(ev); - if (seqnumq == dispatchq) { + if (seqnumq == dispatchq || completionq == dispatchq) { ctx = get_pkt_ctx_from_pkt(pkt); } else { ctx = alloc_pkt_ctx(pkt); @@ -1110,15 +1133,14 @@ int pktio_thread(void *arg EXAMPLE_UNUSED) case PKT_STATE_IPSEC_IN_CLASSIFY: ctx->state = PKT_STATE_ROUTE_LOOKUP; - rc = do_ipsec_in_classify(pkt, + rc = do_ipsec_in_classify(&pkt, ctx, - &skip, - &result); + &skip); break; case PKT_STATE_IPSEC_IN_FINISH: - rc = do_ipsec_in_finish(pkt, ctx, &result); + rc = do_ipsec_in_finish(pkt, ctx); ctx->state = PKT_STATE_ROUTE_LOOKUP; break; @@ -1145,12 +1167,12 @@ int pktio_thread(void *arg EXAMPLE_UNUSED) case PKT_STATE_IPSEC_OUT_SEQ: ctx->state = PKT_STATE_IPSEC_OUT_FINISH; - rc = do_ipsec_out_seq(pkt, ctx, &result); + rc = do_ipsec_out_seq(&pkt, ctx); break; case PKT_STATE_IPSEC_OUT_FINISH: - rc = do_ipsec_out_finish(pkt, ctx, &result); + rc = do_ipsec_out_finish(pkt, ctx); ctx->state = PKT_STATE_TRANSMIT; break; diff --git a/example/ipsec/odp_ipsec_cache.c b/example/ipsec/odp_ipsec_cache.c index e4150336..18a98a29 100644 --- a/example/ipsec/odp_ipsec_cache.c +++ b/example/ipsec/odp_ipsec_cache.c @@ -40,7 +40,9 @@ int create_ipsec_cache_entry(sa_db_entry_t *cipher_sa, sa_db_entry_t *auth_sa, tun_db_entry_t *tun, crypto_api_mode_e api_mode, - odp_bool_t in) + odp_bool_t in, + odp_queue_t completionq, + odp_pool_t out_pool) { odp_crypto_session_param_t params; ipsec_cache_entry_t *entry; @@ -63,8 +65,17 @@ int create_ipsec_cache_entry(sa_db_entry_t *cipher_sa, /* Setup parameters and call crypto library to create session */ params.op = (in) ? ODP_CRYPTO_OP_DECODE : ODP_CRYPTO_OP_ENCODE; params.auth_cipher_text = TRUE; - params.compl_queue = ODP_QUEUE_INVALID; - params.output_pool = ODP_POOL_INVALID; + if (CRYPTO_API_SYNC == api_mode) { + params.packet_op_mode = ODP_CRYPTO_SYNC; + params.compl_queue = ODP_QUEUE_INVALID; + params.output_pool = ODP_POOL_INVALID; + entry->async = FALSE; + } else { + params.packet_op_mode = ODP_CRYPTO_ASYNC; + params.compl_queue = completionq; + params.output_pool = out_pool; + entry->async = TRUE; + } if (CRYPTO_API_ASYNC_NEW_BUFFER == api_mode) entry->in_place = FALSE; diff --git a/example/ipsec/odp_ipsec_cache.h b/example/ipsec/odp_ipsec_cache.h index ce37ccce..45010249 100644 --- a/example/ipsec/odp_ipsec_cache.h +++ b/example/ipsec/odp_ipsec_cache.h @@ -32,6 +32,7 @@ typedef enum { typedef struct ipsec_cache_entry_s { struct ipsec_cache_entry_s *next; /**< Next entry on list */ odp_bool_t in_place; /**< Crypto API mode */ + odp_bool_t async; /**< ASYNC or SYNC mode */ uint32_t src_ip; /**< Source v4 address */ uint32_t dst_ip; /**< Destination v4 address */ sa_mode_t mode; /**< SA mode - transport/tun */ @@ -85,6 +86,8 @@ void init_ipsec_cache(void); * @param tun Tunnel DB entry pointer * @param api_mode Crypto API mode for testing * @param in Direction (input versus output) + * @param completionq Completion queue + * @param out_pool Output buffer pool * * @return 0 if successful else -1 */ @@ -92,7 +95,9 @@ int create_ipsec_cache_entry(sa_db_entry_t *cipher_sa, sa_db_entry_t *auth_sa, tun_db_entry_t *tun, crypto_api_mode_e api_mode, - odp_bool_t in); + odp_bool_t in, + odp_queue_t completionq, + odp_pool_t out_pool); /** * Find a matching IPsec cache entry for input packet diff --git a/example/ipsec/odp_ipsec_misc.h b/example/ipsec/odp_ipsec_misc.h index 20ebe9fc..346cc1c2 100644 --- a/example/ipsec/odp_ipsec_misc.h +++ b/example/ipsec/odp_ipsec_misc.h @@ -321,7 +321,7 @@ void ipv4_adjust_len(odph_ipv4hdr_t *ip, int adj) /** * Verify crypto operation completed successfully * - * @param status Pointer to crypto op status structure + * @param status Pointer to cryto completion structure * * @return TRUE if all OK else FALSE */ commit 533a2fe155aa16268bfe4091a3a056a78d5dbfc6 Author: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Date: Fri Jun 30 18:33:28 2017 +0300 test: rewrite odp_crypto using Crypto packet API Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Petri Savolainen <petri.savolainen(a)nokia.com> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/test/common_plat/performance/odp_crypto.c b/test/common_plat/performance/odp_crypto.c index bc29a888..a129ec98 100644 --- a/test/common_plat/performance/odp_crypto.c +++ b/test/common_plat/performance/odp_crypto.c @@ -443,9 +443,10 @@ create_session_from_config(odp_crypto_session_t *session, return -1; } params.compl_queue = out_queue; - + params.packet_op_mode = ODP_CRYPTO_ASYNC; } else { params.compl_queue = ODP_QUEUE_INVALID; + params.packet_op_mode = ODP_CRYPTO_SYNC; } if (odp_crypto_session_create(&params, session, &ses_create_rc)) { @@ -456,6 +457,24 @@ create_session_from_config(odp_crypto_session_t *session, return 0; } +static odp_packet_t +make_packet(odp_pool_t pkt_pool, unsigned int payload_length) +{ + odp_packet_t pkt; + + pkt = odp_packet_alloc(pkt_pool, payload_length); + if (pkt == ODP_PACKET_INVALID) { + app_err("failed to allocate buffer\n"); + return pkt; + } + + void *mem = odp_packet_data(pkt); + + memset(mem, 1, payload_length); + + return pkt; +} + /** * Run measurement iterations for given config and payload size. * Result of run returned in 'result' out parameter. @@ -467,15 +486,13 @@ run_measure_one(crypto_args_t *cargs, unsigned int payload_length, crypto_run_result_t *result) { - odp_crypto_op_param_t params; + odp_crypto_packet_op_param_t params; odp_pool_t pkt_pool; odp_queue_t out_queue; - odp_packet_t pkt; + odp_packet_t pkt = ODP_PACKET_INVALID; int rc = 0; - odp_bool_t posted = 0; - pkt_pool = odp_pool_lookup("packet_pool"); if (pkt_pool == ODP_POOL_INVALID) { app_err("pkt_pool not found\n"); @@ -490,16 +507,12 @@ run_measure_one(crypto_args_t *cargs, } } - pkt = odp_packet_alloc(pkt_pool, payload_length); - if (pkt == ODP_PACKET_INVALID) { - app_err("failed to allocate buffer\n"); - return -1; + if (cargs->reuse_packet) { + pkt = make_packet(pkt_pool, payload_length); + if (ODP_PACKET_INVALID == pkt) + return -1; } - void *mem = odp_packet_data(pkt); - - memset(mem, 1, payload_length); - time_record_t start, end; int packets_sent = 0; int packets_received = 0; @@ -515,77 +528,102 @@ run_measure_one(crypto_args_t *cargs, params.auth_range.length = payload_length; params.hash_result_offset = payload_length; - if (cargs->reuse_packet) { - params.pkt = pkt; - params.out_pkt = cargs->in_place ? pkt : - ODP_PACKET_INVALID; - } - fill_time_record(&start); while ((packets_sent < cargs->iteration_count) || (packets_received < cargs->iteration_count)) { void *mem; - odp_crypto_op_result_t result; if ((packets_sent < cargs->iteration_count) && (packets_sent - packets_received < cargs->in_flight)) { + odp_packet_t out_pkt; + if (!cargs->reuse_packet) { - /* - * For in place test we use just one - * statically allocated buffer. - * For now in place test we have to - * allocate and initialize packet - * every time. - * Note we leaked one packet here. - */ - odp_packet_t newpkt; - - newpkt = odp_packet_alloc(pkt_pool, - payload_length); - if (newpkt == ODP_PACKET_INVALID) { - app_err("failed to allocate buffer\n"); + pkt = make_packet(pkt_pool, payload_length); + if (ODP_PACKET_INVALID == pkt) return -1; - } - mem = odp_packet_data(newpkt); - memset(mem, 1, payload_length); - params.pkt = newpkt; - params.out_pkt = cargs->in_place ? newpkt : - ODP_PACKET_INVALID; } + out_pkt = cargs->in_place ? pkt : ODP_PACKET_INVALID; + if (cargs->debug_packets) { - mem = odp_packet_data(params.pkt); + mem = odp_packet_data(pkt); print_mem("Packet before encryption:", mem, payload_length); } - rc = odp_crypto_operation(&params, &posted, - &result); - if (rc) - app_err("failed odp_crypto_operation: rc = %d\n", - rc); - else - packets_sent++; + if (cargs->schedule || cargs->poll) { + rc = odp_crypto_packet_op_enq(&pkt, &out_pkt, + &params, 1); + if (rc <= 0) { + app_err("failed odp_crypto_packet_op_enq: rc = %d\n", + rc); + break; + } + packets_sent += rc; + } else { + rc = odp_crypto_packet_op(&pkt, &out_pkt, + &params, 1); + if (rc <= 0) { + app_err("failed odp_crypto_packet_op: rc = %d\n", + rc); + break; + } + packets_sent += rc; + packets_received++; + if (cargs->debug_packets) { + mem = odp_packet_data(out_pkt); + print_mem("Immediately encrypted " + "packet", + mem, + payload_length + + config->session. + auth_digest_len); + } + if (!cargs->in_place) { + if (cargs->reuse_packet) + pkt = out_pkt; + else + odp_packet_free(out_pkt); + } + } } - if (1) { - packets_received++; - if (cargs->debug_packets) { - mem = odp_packet_data(params.out_pkt); - print_mem("Immediately encrypted packet", mem, - payload_length + - config->session.auth_digest_len); - } - if (!cargs->in_place) { - if (cargs->reuse_packet) { - params.pkt = params.out_pkt; - params.out_pkt = ODP_PACKET_INVALID; - } else { - odp_packet_free(params.out_pkt); + if (out_queue != ODP_QUEUE_INVALID) { + odp_event_t ev; + odp_crypto_packet_result_t result; + odp_packet_t out_pkt; + + if (cargs->schedule) + ev = odp_schedule(NULL, + ODP_SCHED_NO_WAIT); + else + ev = odp_queue_deq(out_queue); + + while (ev != ODP_EVENT_INVALID) { + out_pkt = odp_crypto_packet_from_event(ev); + odp_crypto_packet_result(&result, out_pkt); + + if (cargs->debug_packets) { + mem = odp_packet_data(out_pkt); + print_mem("Receieved encrypted packet", + mem, + payload_length + + config-> + session.auth_digest_len); } - } + if (cargs->reuse_packet) + pkt = out_pkt; + else + odp_packet_free(out_pkt); + packets_received++; + if (cargs->schedule) + ev = odp_schedule(NULL, + ODP_SCHED_NO_WAIT); + else + ev = odp_queue_deq(out_queue); + }; } } @@ -607,9 +645,10 @@ run_measure_one(crypto_args_t *cargs, cargs->iteration_count; } - odp_packet_free(pkt); + if (ODP_PACKET_INVALID != pkt) + odp_packet_free(pkt); - return rc; + return rc < 0 ? rc : 0; } /** commit b4edd15ea49b38a0c1ed2bd021ee36aa296a79b0 Author: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Date: Sat Jul 1 00:10:34 2017 +0300 validation: crypto: tests for Crypto packet API Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Petri Savolainen <petri.savolainen(a)nokia.com> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/test/common_plat/validation/api/crypto/crypto.c b/test/common_plat/validation/api/crypto/crypto.c index fde95ab7..8a6c20ec 100644 --- a/test/common_plat/validation/api/crypto/crypto.c +++ b/test/common_plat/validation/api/crypto/crypto.c @@ -13,8 +13,12 @@ #define PKT_POOL_LEN (1 * 1024) odp_suiteinfo_t crypto_suites[] = { - {ODP_CRYPTO_SYNC_INP, crypto_suite_sync_init, crypto_suite_term, - crypto_suite}, + {ODP_CRYPTO_SYNC_INP, crypto_suite_sync_init, + crypto_suite_term, crypto_suite}, + {ODP_CRYPTO_PACKET_SYNC_INP, crypto_suite_packet_sync_init, + crypto_suite_term, crypto_suite}, + {ODP_CRYPTO_PACKET_ASYNC_INP, crypto_suite_packet_async_init, + crypto_suite_term, crypto_suite}, ODP_SUITE_INFO_NULL, }; diff --git a/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c b/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c index d42ecb68..d5549493 100644 --- a/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c +++ b/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c @@ -14,6 +14,8 @@ #define MAX_ALG_CAPA 32 struct suite_context_s { + odp_bool_t packet; + odp_crypto_op_mode_t packet_op_mode; odp_pool_t pool; odp_queue_t queue; }; @@ -58,6 +60,217 @@ static const char *cipher_alg_name(odp_cipher_alg_t cipher) } } +static int alg_op(odp_packet_t pkt, + odp_bool_t *ok, + odp_crypto_session_t session, + uint8_t *op_iv_ptr, + odp_packet_data_range_t *cipher_range, + odp_packet_data_range_t *auth_range, + uint8_t *aad, + uint32_t aad_len, + unsigned int plaintext_len) +{ + int data_off = 0, rc; + odp_crypto_op_result_t result; + odp_crypto_op_param_t op_params; + odp_bool_t posted; + odp_event_subtype_t subtype; + + /* Prepare input/output params */ + memset(&op_params, 0, sizeof(op_params)); + op_params.session = session; + op_params.pkt = pkt; + op_params.out_pkt = pkt; + + if (cipher_range) { + op_params.cipher_range = *cipher_range; + data_off = cipher_range->offset; + } else { + op_params.cipher_range.offset = data_off; + op_params.cipher_range.length = plaintext_len; + } + if (auth_range) { + op_params.auth_range = *auth_range; + } else { + op_params.auth_range.offset = data_off; + op_params.auth_range.length = plaintext_len; + } + if (op_iv_ptr) + op_params.override_iv_ptr = op_iv_ptr; + + op_params.aad.ptr = aad; + op_params.aad.length = aad_len; + + op_params.hash_result_offset = plaintext_len; + + rc = odp_crypto_operation(&op_params, &posted, &result); + if (rc < 0) { + CU_FAIL("Failed odp_crypto_operation()"); + return rc; + } + + CU_ASSERT(posted == 0); + CU_ASSERT(result.pkt == pkt); + CU_ASSERT(ODP_EVENT_PACKET == + odp_event_type(odp_packet_to_event(result.pkt))); + CU_ASSERT(ODP_EVENT_PACKET_BASIC == + odp_event_subtype(odp_packet_to_event(result.pkt))); + CU_ASSERT(ODP_EVENT_PACKET == + odp_event_types(odp_packet_to_event(result.pkt), &subtype)); + CU_ASSERT(ODP_EVENT_PACKET_BASIC == subtype); + + *ok = result.ok; + + return 0; +} + +static int alg_packet_op(odp_packet_t pkt, + odp_bool_t *ok, + odp_crypto_session_t session, + uint8_t *op_iv_ptr, + odp_packet_data_range_t *cipher_range, + odp_packet_data_range_t *auth_range, + uint8_t *aad, + uint32_t aad_len, + unsigned int plaintext_len) +{ + int data_off = 0, rc; + odp_crypto_packet_result_t result; + odp_crypto_packet_op_param_t op_params; + odp_event_subtype_t subtype; + odp_packet_t out_pkt = pkt; + + /* Prepare input/output params */ + memset(&op_params, 0, sizeof(op_params)); + op_params.session = session; + + if (cipher_range) { + op_params.cipher_range = *cipher_range; + data_off = cipher_range->offset; + } else { + op_params.cipher_range.offset = data_off; + op_params.cipher_range.length = plaintext_len; + } + if (auth_range) { + op_params.auth_range = *auth_range; + } else { + op_params.auth_range.offset = data_off; + op_params.auth_range.length = plaintext_len; + } + if (op_iv_ptr) + op_params.override_iv_ptr = op_iv_ptr; + + op_params.aad.ptr = aad; + op_params.aad.length = aad_len; + + op_params.hash_result_offset = plaintext_len; + + rc = odp_crypto_packet_op(&pkt, &out_pkt, &op_params, 1); + if (rc < 0) { + CU_FAIL("Failed odp_crypto_packet_op()"); + return rc; + } + + CU_ASSERT(out_pkt == pkt); + CU_ASSERT(ODP_EVENT_PACKET == + odp_event_type(odp_packet_to_event(pkt))); + CU_ASSERT(ODP_EVENT_PACKET_CRYPTO == + odp_event_subtype(odp_packet_to_event(pkt))); + CU_ASSERT(ODP_EVENT_PACKET == + odp_event_types(odp_packet_to_event(pkt), &subtype)); + CU_ASSERT(ODP_EVENT_PACKET_CRYPTO == subtype); + + rc = odp_crypto_packet_result(&result, pkt); + if (rc < 0) { + CU_FAIL("Failed odp_crypto_packet_result()"); + return rc; + } + + *ok = result.ok; + + return 0; +} + +static int alg_packet_op_enq(odp_packet_t pkt, + odp_bool_t *ok, + odp_crypto_session_t session, + uint8_t *op_iv_ptr, + odp_packet_data_range_t *cipher_range, + odp_packet_data_range_t *auth_range, + uint8_t *aad, + uint32_t aad_len, + unsigned int plaintext_len) +{ + int data_off = 0, rc; + odp_event_t event; + odp_crypto_packet_result_t result; + odp_crypto_packet_op_param_t op_params; + odp_event_subtype_t subtype; + odp_packet_t out_pkt = pkt; + + /* Prepare input/output params */ + memset(&op_params, 0, sizeof(op_params)); + op_params.session = session; + + if (cipher_range) { + op_params.cipher_range = *cipher_range; + data_off = cipher_range->offset; + } else { + op_params.cipher_range.offset = data_off; + op_params.cipher_range.length = plaintext_len; + } + if (auth_range) { + op_params.auth_range = *auth_range; + } else { + op_params.auth_range.offset = data_off; + op_params.auth_range.length = plaintext_len; + } + if (op_iv_ptr) + op_params.override_iv_ptr = op_iv_ptr; + + op_params.aad.ptr = aad; + op_params.aad.length = aad_len; + + op_params.hash_result_offset = plaintext_len; + + rc = odp_crypto_packet_op_enq(&pkt, &pkt, &op_params, 1); + if (rc < 0) { + CU_FAIL("Failed odp_crypto_op_enq()"); + return rc; + } + + /* Poll completion queue for results */ + do { + event = odp_queue_deq(suite_context.queue); + } while (event == ODP_EVENT_INVALID); + + CU_ASSERT(ODP_EVENT_PACKET == odp_event_type(event)); + CU_ASSERT(ODP_EVENT_PACKET_CRYPTO == odp_event_subtype(event)); + CU_ASSERT(ODP_EVENT_PACKET == odp_event_types(event, &subtype)); + CU_ASSERT(ODP_EVENT_PACKET_CRYPTO == subtype); + + pkt = odp_crypto_packet_from_event(event); + + CU_ASSERT(out_pkt == pkt); + CU_ASSERT(ODP_EVENT_PACKET == + odp_event_type(odp_packet_to_event(pkt))); + CU_ASSERT(ODP_EVENT_PACKET_CRYPTO == + odp_event_subtype(odp_packet_to_event(pkt))); + CU_ASSERT(ODP_EVENT_PACKET == + odp_event_types(odp_packet_to_event(pkt), &subtype)); + CU_ASSERT(ODP_EVENT_PACKET_CRYPTO == subtype); + + rc = odp_crypto_packet_result(&result, pkt); + if (rc < 0) { + CU_FAIL("Failed odp_crypto_packet_result()"); + return rc; + } + + *ok = result.ok; + + return 0; +} + /* Basic algorithm run function for async inplace mode. * Creates a session from input parameters and runs one operation * on input_vec. Checks the output of the crypto operation against @@ -90,17 +303,13 @@ static void alg_test(odp_crypto_op_t op, odp_crypto_capability_t capa; int rc; odp_crypto_ses_create_err_t status; - odp_bool_t posted; - odp_crypto_op_result_t result; + odp_bool_t ok; odp_crypto_session_param_t ses_params; - odp_crypto_op_param_t op_params; uint8_t *data_addr; - int data_off; odp_crypto_cipher_capability_t cipher_capa[MAX_ALG_CAPA]; odp_crypto_auth_capability_t auth_capa[MAX_ALG_CAPA]; int num, i; int found; - odp_event_subtype_t subtype; rc = odp_crypto_capability(&capa); CU_ASSERT(!rc); @@ -193,6 +402,7 @@ static void alg_test(odp_crypto_op_t op, odp_crypto_session_param_init(&ses_params); ses_params.op = op; ses_params.auth_cipher_text = false; + ses_params.packet_op_mode = suite_context.packet_op_mode; ses_params.cipher_alg = cipher_alg; ses_params.auth_alg = auth_alg; ses_params.compl_queue = suite_context.queue; @@ -214,67 +424,40 @@ static void alg_test(odp_crypto_op_t op, CU_ASSERT(pkt != ODP_PACKET_INVALID); data_addr = odp_packet_data(pkt); memcpy(data_addr, plaintext, plaintext_len); - data_off = 0; - - /* Prepare input/output params */ - memset(&op_params, 0, sizeof(op_params)); - op_params.session = session; - op_params.pkt = pkt; - op_params.out_pkt = pkt; - - if (cipher_range) { - op_params.cipher_range = *cipher_range; - data_off = cipher_range->offset; - } else { - op_params.cipher_range.offset = data_off; - op_params.cipher_range.length = plaintext_len; - } - if (auth_range) { - op_params.auth_range = *auth_range; - } else { - op_params.auth_range.offset = data_off; - op_params.auth_range.length = plaintext_len; - } - if (op_iv_ptr) - op_params.override_iv_ptr = op_iv_ptr; - - op_params.aad.ptr = aad; - op_params.aad.length = aad_len; - - op_params.hash_result_offset = plaintext_len; if (0 != digest_len) { - memcpy(data_addr + op_params.hash_result_offset, + memcpy(data_addr + plaintext_len, digest, digest_len); } - rc = odp_crypto_operation(&op_params, &posted, &result); + if (!suite_context.packet) + rc = alg_op(pkt, &ok, session, op_iv_ptr, + cipher_range, auth_range, aad, aad_len, + plaintext_len); + else if (ODP_CRYPTO_ASYNC == suite_context.packet_op_mode) + rc = alg_packet_op_enq(pkt, &ok, session, op_iv_ptr, + cipher_range, auth_range, aad, aad_len, + plaintext_len); + else + rc = alg_packet_op(pkt, &ok, session, op_iv_ptr, + cipher_range, auth_range, aad, aad_len, + plaintext_len); if (rc < 0) { - CU_FAIL("Failed odp_crypto_operation()"); goto cleanup; } - CU_ASSERT(posted == 0); - CU_ASSERT(result.pkt == pkt); - CU_ASSERT(ODP_EVENT_PACKET == - odp_event_type(odp_packet_to_event(result.pkt))); - CU_ASSERT(ODP_EVENT_PACKET_BASIC == - odp_event_subtype(odp_packet_to_event(result.pkt))); - CU_ASSERT(ODP_EVENT_PACKET == - odp_event_types(odp_packet_to_event(result.pkt), &subtype)); - CU_ASSERT(ODP_EVENT_PACKET_BASIC == subtype); - if (should_fail) { - CU_ASSERT(!result.ok); + CU_ASSERT(!ok); goto cleanup; } - CU_ASSERT(result.ok); + CU_ASSERT(ok); + data_addr = odp_packet_data(pkt); if (cipher_alg != ODP_CIPHER_ALG_NULL) CU_ASSERT(!memcmp(data_addr, ciphertext, ciphertext_len)); if (op == ODP_CRYPTO_OP_ENCODE && auth_alg != ODP_AUTH_ALG_NULL) - CU_ASSERT(!memcmp(data_addr + op_params.hash_result_offset, + CU_ASSERT(!memcmp(data_addr + plaintext_len, digest, digest_len)); cleanup: rc = odp_crypto_session_destroy(session); @@ -1498,6 +1681,34 @@ int crypto_suite_sync_init(void) return 0; } +int crypto_suite_packet_sync_init(void) +{ + suite_context.packet = true; + suite_context.packet_op_mode = ODP_CRYPTO_SYNC; + + suite_context.pool = odp_pool_lookup("packet_pool"); + if (suite_context.pool == ODP_POOL_INVALID) + return -1; + + suite_context.queue = ODP_QUEUE_INVALID; + return 0; +} + +int crypto_suite_packet_async_init(void) +{ + suite_context.packet = true; + suite_context.packet_op_mode = ODP_CRYPTO_ASYNC; + + suite_context.pool = odp_pool_lookup("packet_pool"); + if (suite_context.pool == ODP_POOL_INVALID) + return -1; + + suite_context.queue = odp_queue_lookup("crypto-out"); + if (suite_context.queue == ODP_QUEUE_INVALID) + return -1; + return 0; +} + odp_testinfo_t crypto_suite[] = { ODP_TEST_INFO_CONDITIONAL(crypto_test_enc_alg_null, check_alg_null), diff --git a/test/common_plat/validation/api/crypto/odp_crypto_test_inp.h b/test/common_plat/validation/api/crypto/odp_crypto_test_inp.h index 86ecce8e..2bc77a94 100644 --- a/test/common_plat/validation/api/crypto/odp_crypto_test_inp.h +++ b/test/common_plat/validation/api/crypto/odp_crypto_test_inp.h @@ -9,12 +9,16 @@ #include <odp_cunit_common.h> /* Suite names */ -#define ODP_CRYPTO_SYNC_INP "odp_crypto_sync_inp" +#define ODP_CRYPTO_SYNC_INP "odp_crypto_sync_inp" +#define ODP_CRYPTO_PACKET_SYNC_INP "odp_crypto_packet_sync_inp" +#define ODP_CRYPTO_PACKET_ASYNC_INP "odp_crypto_packet_async_inp" /* Suite test array */ extern odp_testinfo_t crypto_suite[]; int crypto_suite_sync_init(void); +int crypto_suite_packet_sync_init(void); +int crypto_suite_packet_async_init(void); int crypto_suite_term(void); #endif commit 99b74cbb3c77902322933192ce68f8c1abe21e84 Author: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Date: Sat Jul 1 00:09:21 2017 +0300 linux-gen: crypto: provide packet API implementation Provide implementation for new crypto API using packets as first grade objects. Reimplement odp_crypto_operation using new functions. Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Petri Savolainen <petri.savolainen(a)nokia.com> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_crypto_internal.h b/platform/linux-generic/include/odp_crypto_internal.h index 52b94d07..12d1720b 100644 --- a/platform/linux-generic/include/odp_crypto_internal.h +++ b/platform/linux-generic/include/odp_crypto_internal.h @@ -23,7 +23,8 @@ typedef struct odp_crypto_generic_session odp_crypto_generic_session_t; * Algorithm handler function prototype */ typedef -odp_crypto_alg_err_t (*crypto_func_t)(odp_crypto_op_param_t *param, +odp_crypto_alg_err_t (*crypto_func_t)(odp_packet_t pkt, + const odp_crypto_packet_op_param_t *param, odp_crypto_generic_session_t *session); /** diff --git a/platform/linux-generic/include/odp_packet_internal.h b/platform/linux-generic/include/odp_packet_internal.h index b74e0cb1..5e0f3204 100644 --- a/platform/linux-generic/include/odp_packet_internal.h +++ b/platform/linux-generic/include/odp_packet_internal.h @@ -129,6 +129,9 @@ typedef struct { /* Classifier destination queue */ queue_t dst_queue; + /* Result for crypto packet op */ + odp_crypto_packet_result_t crypto_op_result; + /* Packet data storage */ uint8_t data[0]; } odp_packet_hdr_t; diff --git a/platform/linux-generic/odp_crypto.c b/platform/linux-generic/odp_crypto.c index 075bb02b..792c2eca 100644 --- a/platform/linux-generic/odp_crypto.c +++ b/platform/linux-generic/odp_crypto.c @@ -113,7 +113,8 @@ void free_session(odp_crypto_generic_session_t *session) } static odp_crypto_alg_err_t -null_crypto_routine(odp_crypto_op_param_t *param ODP_UNUSED, +null_crypto_routine(odp_packet_t pkt ODP_UNUSED, + const odp_crypto_packet_op_param_t *param ODP_UNUSED, odp_crypto_generic_session_t *session ODP_UNUSED) { return ODP_CRYPTO_ALG_ERR_NONE; @@ -121,11 +122,11 @@ null_crypto_routine(odp_crypto_op_param_t *param ODP_UNUSED, static void packet_hmac_calculate(HMAC_CTX *ctx, - odp_crypto_op_param_t *param, + odp_packet_t pkt, + const odp_crypto_packet_op_param_t *param, odp_crypto_generic_session_t *session, uint8_t *hash) { - odp_packet_t pkt = param->out_pkt; uint32_t offset = param->auth_range.offset; uint32_t len = param->auth_range.length; @@ -152,7 +153,8 @@ void packet_hmac_calculate(HMAC_CTX *ctx, #if OPENSSL_VERSION_NUMBER < 0x10100000L static -void packet_hmac(odp_crypto_op_param_t *param, +void packet_hmac(odp_packet_t pkt, + const odp_crypto_packet_op_param_t *param, odp_crypto_generic_session_t *session, uint8_t *hash) { @@ -160,12 +162,13 @@ void packet_hmac(odp_crypto_op_param_t *param, /* Hash it */ HMAC_CTX_init(&ctx); - packet_hmac_calculate(&ctx, param, session, hash); + packet_hmac_calculate(&ctx, pkt, param, session, hash); HMAC_CTX_cleanup(&ctx); } #else static -void packet_hmac(odp_crypto_op_param_t *param, +void packet_hmac(odp_packet_t pkt, + const odp_crypto_packet_op_param_t *param, odp_crypto_generic_session_t *session, uint8_t *hash) { @@ -173,22 +176,23 @@ void packet_hmac(odp_crypto_op_param_t *param, /* Hash it */ ctx = HMAC_CTX_new(); - packet_hmac_calculate(ctx, param, session, hash); + packet_hmac_calculate(ctx, pkt, param, session, hash); HMAC_CTX_free(ctx); } #endif static -odp_crypto_alg_err_t auth_gen(odp_crypto_op_param_t *param, +odp_crypto_alg_err_t auth_gen(odp_packet_t pkt, + const odp_crypto_packet_op_param_t *param, odp_crypto_generic_session_t *session) { uint8_t hash[EVP_MAX_MD_SIZE]; /* Hash it */ - packet_hmac(param, session, hash); + packet_hmac(pkt, param, session, hash); /* Copy to the output location */ - odp_packet_copy_from_mem(param->out_pkt, + odp_packet_copy_from_mem(pkt, param->hash_result_offset, session->auth.bytes, hash); @@ -197,7 +201,8 @@ odp_crypto_alg_err_t auth_gen(odp_crypto_op_param_t *param, } static -odp_crypto_alg_err_t auth_check(odp_crypto_op_param_t *param, +odp_crypto_alg_err_t auth_check(odp_packet_t pkt, + const odp_crypto_packet_op_param_t *param, odp_crypto_generic_session_t *session) { uint32_t bytes = session->auth.bytes; @@ -205,14 +210,14 @@ odp_crypto_alg_err_t auth_check(odp_crypto_op_param_t *param, uint8_t hash_out[EVP_MAX_MD_SIZE]; /* Copy current value out and clear it before authentication */ - odp_packet_copy_to_mem(param->out_pkt, param->hash_result_offset, + odp_packet_copy_to_mem(pkt, param->hash_result_offset, bytes, hash_in); - _odp_packet_set_data(param->out_pkt, param->hash_result_offset, + _odp_packet_set_data(pkt, param->hash_result_offset, 0, bytes); /* Hash it */ - packet_hmac(param, session, hash_out); + packet_hmac(pkt, param, session, hash_out); /* Verify match */ if (0 != memcmp(hash_in, hash_out, bytes)) @@ -223,9 +228,10 @@ odp_crypto_alg_err_t auth_check(odp_crypto_op_param_t *param, } static -int internal_encrypt(EVP_CIPHER_CTX *ctx, odp_crypto_op_param_t *param) +int internal_encrypt(EVP_CIPHER_CTX *ctx, + odp_packet_t pkt, + const odp_crypto_packet_op_param_t *param) { - odp_packet_t pkt = param->out_pkt; unsigned in_pos = param->cipher_range.offset; unsigned out_pos = param->cipher_range.offset; unsigned in_len = param->cipher_range.length; @@ -278,9 +284,10 @@ int internal_encrypt(EVP_CIPHER_CTX *ctx, odp_crypto_op_param_t *param) } static -int internal_decrypt(EVP_CIPHER_CTX *ctx, odp_crypto_op_param_t *param) +int internal_decrypt(EVP_CIPHER_CTX *ctx, + odp_packet_t pkt, + const odp_crypto_packet_op_param_t *param) { - odp_packet_t pkt = param->out_pkt; unsigned in_pos = param->cipher_range.offset; unsigned out_pos = param->cipher_range.offset; unsigned in_len = param->cipher_range.length; @@ -333,7 +340,8 @@ int internal_decrypt(EVP_CIPHER_CTX *ctx, odp_crypto_op_param_t *param) } static -odp_crypto_alg_err_t cipher_encrypt(odp_crypto_op_param_t *param, +odp_crypto_alg_err_t cipher_encrypt(odp_packet_t pkt, + const odp_crypto_packet_op_param_t *param, odp_crypto_generic_session_t *session) { EVP_CIPHER_CTX *ctx; @@ -354,7 +362,7 @@ odp_crypto_alg_err_t cipher_encrypt(odp_crypto_op_param_t *param, EVP_EncryptInit_ex(ctx, NULL, NULL, NULL, iv_ptr); EVP_CIPHER_CTX_set_padding(ctx, 0); - ret = internal_encrypt(ctx, param); + ret = internal_encrypt(ctx, pkt, param); EVP_CIPHER_CTX_free(ctx); @@ -363,7 +371,8 @@ odp_crypto_alg_err_t cipher_encrypt(odp_crypto_op_param_t *param, } static -odp_crypto_alg_err_t cipher_decrypt(odp_crypto_op_param_t *param, +odp_crypto_alg_err_t cipher_decrypt(odp_packet_t pkt, + const odp_crypto_packet_op_param_t *param, odp_crypto_generic_session_t *session) { EVP_CIPHER_CTX *ctx; @@ -384,7 +393,7 @@ odp_crypto_alg_err_t cipher_decrypt(odp_crypto_op_param_t *param, EVP_DecryptInit_ex(ctx, NULL, NULL, NULL, iv_ptr); EVP_CIPHER_CTX_set_padding(ctx, 0); - ret = internal_decrypt(ctx, param); + ret = internal_decrypt(ctx, pkt, param); EVP_CIPHER_CTX_free(ctx); @@ -420,7 +429,8 @@ static int process_cipher_param(odp_crypto_generic_session_t *session, } static -odp_crypto_alg_err_t aes_gcm_encrypt(odp_crypto_op_param_t *param, +odp_crypto_alg_err_t aes_gcm_encrypt(odp_packet_t pkt, + const odp_crypto_packet_op_param_t *param, odp_crypto_generic_session_t *session) { EVP_CIPHER_CTX *ctx; @@ -452,11 +462,11 @@ odp_crypto_alg_err_t aes_gcm_encrypt(odp_crypto_op_param_t *param, EVP_EncryptUpdate(ctx, NULL, &dummy_len, aad_head, aad_len); - ret = internal_encrypt(ctx, param); + ret = internal_encrypt(ctx, pkt, param); EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_GET_TAG, session->p.auth_digest_len, block); - odp_packet_copy_from_mem(param->out_pkt, param->hash_result_offset, + odp_packet_copy_from_mem(pkt, param->hash_result_offset, session->p.auth_digest_len, block); EVP_CIPHER_CTX_free(ctx); @@ -466,7 +476,8 @@ odp_crypto_alg_err_t aes_gcm_encrypt(odp_crypto_op_param_t *param, } static -odp_crypto_alg_err_t aes_gcm_decrypt(odp_crypto_op_param_t *param, +odp_crypto_alg_err_t aes_gcm_decrypt(odp_packet_t pkt, + const odp_crypto_packet_op_param_t *param, odp_crypto_generic_session_t *session) { EVP_CIPHER_CTX *ctx; @@ -493,7 +504,7 @@ odp_crypto_alg_err_t aes_gcm_decrypt(odp_crypto_op_param_t *param, EVP_DecryptInit_ex(ctx, NULL, NULL, NULL, iv_ptr); EVP_CIPHER_CTX_set_padding(ctx, 0); - odp_packet_copy_to_mem(param->out_pkt, param->hash_result_offset, + odp_packet_copy_to_mem(pkt, param->hash_result_offset, session->p.auth_digest_len, block); EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_TAG, session->p.auth_digest_len, block); @@ -503,7 +514,7 @@ odp_crypto_alg_err_t aes_gcm_decrypt(odp_crypto_op_param_t *param, EVP_DecryptUpdate(ctx, NULL, &dummy_len, aad_head, aad_len); - ret = internal_decrypt(ctx, param); + ret = internal_decrypt(ctx, pkt, param); EVP_CIPHER_CTX_free(ctx); @@ -833,87 +844,59 @@ int odp_crypto_session_destroy(odp_crypto_session_t session) return 0; } +/* + * Shim function around packet operation, can be used by other implementations. + */ int odp_crypto_operation(odp_crypto_op_param_t *param, odp_bool_t *posted, odp_crypto_op_result_t *result) { - odp_crypto_alg_err_t rc_cipher = ODP_CRYPTO_ALG_ERR_NONE; - odp_crypto_alg_err_t rc_auth = ODP_CRYPTO_ALG_ERR_NONE; - odp_crypto_generic_session_t *session; + odp_crypto_packet_op_param_t packet_param; + odp_packet_t out_pkt = param->out_pkt; + odp_crypto_packet_result_t packet_result; odp_crypto_op_result_t local_result; - odp_bool_t allocated = false; - - session = (odp_crypto_generic_session_t *)(intptr_t)param->session; + int rc; - /* Resolve output buffer */ - if (ODP_PACKET_INVALID == param->out_pkt && - ODP_POOL_INVALID != session->p.output_pool) { - param->out_pkt = odp_packet_alloc(session->p.output_pool, - odp_packet_len(param->pkt)); - allocated = true; - } + packet_param.session = param->session; + packet_param.override_iv_ptr = param->override_iv_ptr; + packet_param.hash_result_offset = param->hash_result_offset; + packet_param.aad.ptr = param->aad.ptr; + packet_param.aad.length = param->aad.length; + packet_param.cipher_range = param->cipher_range; + packet_param.auth_range = param->auth_range; - if (odp_unlikely(ODP_PACKET_INVALID == param->out_pkt)) { - ODP_DBG("Alloc failed.\n"); - return -1; - } + rc = odp_crypto_packet_op(&param->pkt, &out_pkt, &packet_param, 1); + if (rc < 0) + return rc; - if (param->pkt != param->out_pkt) { - int ret; + rc = odp_crypto_packet_result(&packet_result, out_pkt); + if (rc < 0) + return rc; - ret = odp_packet_copy_from_pkt(param->out_pkt, - 0, - param->pkt, - 0, - odp_packet_len(param->pkt)); - if (odp_unlikely(ret < 0)) - goto err; + /* Indicate to caller operation was sync */ + *posted = 0; - _odp_packet_copy_md_to_packet(param->pkt, param->out_pkt); - odp_packet_free(param->pkt); - param->pkt = ODP_PACKET_INVALID; - } - - /* Invoke the functions */ - if (session->do_cipher_first) { - rc_cipher = session->cipher.func(param, session); - rc_auth = session->auth.func(param, session); - } else { - rc_auth = session->auth.func(param, session); - rc_cipher = session->cipher.func(param, session); - } + _odp_buffer_event_subtype_set(_odp_packet_to_buffer(out_pkt), + ODP_EVENT_PACKET_BASIC); /* Fill in result */ #if ODP_DEPRECATED_API local_result.ctx = param->ctx; #endif - local_result.pkt = param->out_pkt; - local_result.cipher_status.alg_err = rc_cipher; - local_result.cipher_status.hw_err = ODP_CRYPTO_HW_ERR_NONE; - local_result.auth_status.alg_err = rc_auth; - local_result.auth_status.hw_err = ODP_CRYPTO_HW_ERR_NONE; - local_result.ok = - (rc_cipher == ODP_CRYPTO_ALG_ERR_NONE) && - (rc_auth == ODP_CRYPTO_ALG_ERR_NONE); + local_result.pkt = out_pkt; + local_result.cipher_status = packet_result.cipher_status; + local_result.auth_status = packet_result.auth_status; + local_result.ok = packet_result.ok; - /* Synchronous, simply return results */ - if (!result) - goto err; - *result = local_result; + /* + * Be bug-to-bug compatible. Return output packet also through params. + */ + param->out_pkt = out_pkt; - /* Indicate to caller operation was sync */ - *posted = 0; + *result = local_result; return 0; - -err: - if (allocated) { - odp_packet_free(param->out_pkt); - param->out_pkt = ODP_PACKET_INVALID; - } - - return -1; } static void ODP_UNUSED openssl_thread_id(CRYPTO_THREADID ODP_UNUSED *id) @@ -1095,3 +1078,171 @@ uint64_t odp_crypto_session_to_u64(odp_crypto_session_t hdl) { return (uint64_t)hdl; } + +odp_packet_t odp_crypto_packet_from_event(odp_event_t ev) +{ + /* This check not mandated by the API specification */ + ODP_ASSERT(odp_event_type(ev) == ODP_EVENT_PACKET); + ODP_ASSERT(odp_event_subtype(ev) == ODP_EVENT_PACKET_CRYPTO); + + return odp_packet_from_event(ev); +} + +odp_event_t odp_crypto_packet_to_event(odp_packet_t pkt) +{ + return odp_packet_to_event(pkt); +} + +static +odp_crypto_packet_result_t *get_op_result_from_packet(odp_packet_t pkt) +{ + odp_packet_hdr_t *hdr = odp_packet_hdr(pkt); + + return &hdr->crypto_op_result; +} + +int odp_crypto_packet_result(odp_crypto_packet_result_t *result, + odp_packet_t packet) +{ + odp_crypto_packet_result_t *op_result; + + ODP_ASSERT(odp_event_subtype(odp_packet_to_event(packet)) == + ODP_EVENT_PACKET_CRYPTO); + + op_result = get_op_result_from_packet(packet); + + memcpy(result, op_result, sizeof(*result)); + + return 0; +} + +static +int odp_crypto_op(odp_packet_t pkt_in, + odp_packet_t *pkt_out, + const odp_crypto_packet_op_param_t *param) +{ + odp_crypto_alg_err_t rc_cipher = ODP_CRYPTO_ALG_ERR_NONE; + odp_crypto_alg_err_t rc_auth = ODP_CRYPTO_ALG_ERR_NONE; + odp_crypto_generic_session_t *session; + odp_crypto_packet_result_t local_result; + odp_bool_t allocated = false; + odp_packet_t out_pkt = *pkt_out; + odp_crypto_packet_result_t *op_result; + + session = (odp_crypto_generic_session_t *)(intptr_t)param->session; + + /* Resolve output buffer */ + if (ODP_PACKET_INVALID == out_pkt && + ODP_POOL_INVALID != session->p.output_pool) { + out_pkt = odp_packet_alloc(session->p.output_pool, + odp_packet_len(pkt_in)); + allocated = true; + } + + if (odp_unlikely(ODP_PACKET_INVALID == out_pkt)) { + ODP_DBG("Alloc failed.\n"); + return -1; + } + + if (pkt_in != out_pkt) { + int ret; + + ret = odp_packet_copy_from_pkt(out_pkt, + 0, + pkt_in, + 0, + odp_packet_len(pkt_in)); + if (odp_unlikely(ret < 0)) + goto err; + + _odp_packet_copy_md_to_packet(pkt_in, out_pkt); + odp_packet_free(pkt_in); + pkt_in = ODP_PACKET_INVALID; + } + + /* Invoke the functions */ + if (session->do_cipher_first) { + rc_cipher = session->cipher.func(out_pkt, param, session); + rc_auth = session->auth.func(out_pkt, param, session); + } else { + rc_auth = session->auth.func(out_pkt, param, session); + rc_cipher = session->cipher.func(out_pkt, param, session); + } + + /* Fill in result */ + local_result.cipher_status.alg_err = rc_cipher; + local_result.cipher_status.hw_err = ODP_CRYPTO_HW_ERR_NONE; + local_result.auth_status.alg_err = rc_auth; + local_result.auth_status.hw_err = ODP_CRYPTO_HW_ERR_NONE; + local_result.ok = + (rc_cipher == ODP_CRYPTO_ALG_ERR_NONE) && + (rc_auth == ODP_CRYPTO_ALG_ERR_NONE); + + _odp_buffer_event_subtype_set(_odp_packet_to_buffer(out_pkt), + ODP_EVENT_PACKET_CRYPTO); + op_result = get_op_result_from_packet(out_pkt); + *op_result = local_result; + + /* Synchronous, simply return results */ + *pkt_out = out_pkt; + + return 0; + +err: + if (allocated) { + odp_packet_free(out_pkt); + out_pkt = ODP_PACKET_INVALID; + } + + return -1; +} + +int odp_crypto_packet_op(const odp_packet_t pkt_in[], + odp_packet_t pkt_out[], + const odp_crypto_packet_op_param_t param[], + int num_pkt) +{ + int i, rc; + odp_crypto_generic_session_t *session; + + session = (odp_crypto_generic_session_t *)(intptr_t)param->session; + ODP_ASSERT(ODP_CRYPTO_SYNC == session->p.packet_op_mode); + + for (i = 0; i < num_pkt; i++) { + rc = odp_crypto_op(pkt_in[i], &pkt_out[i], &param[i]); + if (rc < 0) + break; + } + + return i; +} + +int odp_crypto_packet_op_enq(const odp_packet_t pkt_in[], + const odp_packet_t pkt_out[], + const odp_crypto_packet_op_param_t param[], + int num_pkt) +{ + odp_packet_t pkt; + odp_event_t event; + odp_crypto_generic_session_t *session; + int i, rc; + + session = (odp_crypto_generic_session_t *)(intptr_t)param->session; + ODP_ASSERT(ODP_CRYPTO_ASYNC == session->p.packet_op_mode); + ODP_ASSERT(ODP_QUEUE_INVALID != session->p.compl_queue); + + for (i = 0; i < num_pkt; i++) { + pkt = pkt_out[i]; + rc = odp_crypto_op(pkt_in[i], &pkt, &param[i]); + if (rc < 0) + break; + + event = odp_packet_to_event(pkt); + if (odp_queue_enq(session->p.compl_queue, event)) { + odp_event_free(event); + break; + } + } + + return i; +} commit d0e3acc0cfdab0d4c66b49ea70207d8938569b80 Author: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Date: Wed Jun 21 14:48:00 2017 +0300 api: crypto: add crypto packet operation interface Input and output of crypto operations are packets. API is more flexible for application and API pipelining when output is packet with additional metadata. Application or API pipeline stages which do not care about crypto results may work on basic packet metadata. Crypto result event type changes from ODP_EVENT_CRYPTO_COMPL to ODP_EVENT_PACKET. Event subtype (ODP_EVENT_PACKET_CRYPTO) can be used to identify packets with crypto metadata. Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Petri Savolainen <petri.savolainen(a)nokia.com> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/example/ipsec/odp_ipsec.c b/example/ipsec/odp_ipsec.c index 1ed03585..c618cc46 100644 --- a/example/ipsec/odp_ipsec.c +++ b/example/ipsec/odp_ipsec.c @@ -724,9 +724,9 @@ pkt_disposition_e do_ipsec_in_finish(odp_packet_t pkt, /* Check crypto result */ if (!result->ok) { - if (!is_crypto_compl_status_ok(&result->cipher_status)) + if (!is_crypto_op_status_ok(&result->cipher_status)) return PKT_DROP; - if (!is_crypto_compl_status_ok(&result->auth_status)) + if (!is_crypto_op_status_ok(&result->auth_status)) return PKT_DROP; } ip = (odph_ipv4hdr_t *)odp_packet_l3_ptr(pkt, NULL); @@ -1011,9 +1011,9 @@ pkt_disposition_e do_ipsec_out_finish(odp_packet_t pkt, /* Check crypto result */ if (!result->ok) { - if (!is_crypto_compl_status_ok(&result->cipher_status)) + if (!is_crypto_op_status_ok(&result->cipher_status)) return PKT_DROP; - if (!is_crypto_compl_status_ok(&result->auth_status)) + if (!is_crypto_op_status_ok(&result->auth_status)) return PKT_DROP; } ip = (odph_ipv4hdr_t *)odp_packet_l3_ptr(pkt, NULL); diff --git a/example/ipsec/odp_ipsec_misc.h b/example/ipsec/odp_ipsec_misc.h index 45cb022e..20ebe9fc 100644 --- a/example/ipsec/odp_ipsec_misc.h +++ b/example/ipsec/odp_ipsec_misc.h @@ -321,12 +321,12 @@ void ipv4_adjust_len(odph_ipv4hdr_t *ip, int adj) /** * Verify crypto operation completed successfully * - * @param status Pointer to cryto completion structure + * @param status Pointer to crypto op status structure * * @return TRUE if all OK else FALSE */ static inline -odp_bool_t is_crypto_compl_status_ok(odp_crypto_compl_status_t *status) +odp_bool_t is_crypto_op_status_ok(odp_crypto_op_status_t *status) { if (status->alg_err != ODP_CRYPTO_ALG_ERR_NONE) return FALSE; diff --git a/include/odp/api/spec/crypto.h b/include/odp/api/spec/crypto.h index b9d42937..013507f9 100644 --- a/include/odp/api/spec/crypto.h +++ b/include/odp/api/spec/crypto.h @@ -16,6 +16,7 @@ #include <odp/visibility_begin.h> #include <odp/api/deprecated.h> +#include <odp/api/support.h> #ifdef __cplusplus extern "C" { @@ -276,6 +277,9 @@ typedef struct odp_crypto_session_param_t { * data in non-posted mode */ odp_crypto_op_mode_t ODP_DEPRECATE(pref_mode); + /** Operation mode when using packet interface: sync or async */ + odp_crypto_op_mode_t packet_op_mode; + /** Cipher algorithm * * Use odp_crypto_capability() for supported algorithms. @@ -311,16 +315,15 @@ typedef struct odp_crypto_session_param_t { /** Async mode completion event queue * - * When odp_crypto_operation() is asynchronous, the completion queue is - * used to return the completion status of the operation to the - * application. + * The completion queue is used to return odp_crypto_packet_op_enq() + * results to the application. */ odp_queue_t compl_queue; /** Output pool * * When the output packet is not specified during the call to - * odp_crypto_operation(), the output packet will be allocated + * crypto operation, the output packet will be allocated * from this pool. */ odp_pool_t output_pool; @@ -400,6 +403,44 @@ typedef struct odp_crypto_op_param_t { typedef odp_crypto_op_param_t ODP_DEPRECATE(odp_crypto_op_params_t); /** + * Crypto packet API per packet operation parameters + */ +typedef struct odp_crypto_packet_op_param_t { + /** Session handle from creation */ + odp_crypto_session_t session; + + /** Override session IV pointer */ + uint8_t *override_iv_ptr; + + /** Offset from start of packet for hash result + * + * Specifies the offset where the hash result is to be stored. In case + * of decode sessions, input hash values will be read from this offset, + * and overwritten with hash results. If this offset lies within + * specified 'auth_range', implementation will mute this field before + * calculating the hash result. + */ + uint32_t hash_result_offset; + + /** Additional Authenticated Data (AAD) */ + struct { + /** Pointer to ADD */ + uint8_t *ptr; + + /** AAD length in bytes. Use odp_crypto_auth_capability() for + * supported AAD lengths. */ + uint32_t length; + } aad; + + /** Data range to apply cipher */ + odp_packet_data_range_t cipher_range; + + /** Data range to authenticate */ + odp_packet_data_range_t auth_range; + +} odp_crypto_packet_op_param_t; + +/** * Crypto API session creation return code */ typedef enum { @@ -444,14 +485,17 @@ typedef enum { /** * Cryto API per packet operation completion status */ -typedef struct odp_crypto_compl_status { +typedef struct odp_crypto_op_status { /** Algorithm specific return code */ odp_crypto_alg_err_t alg_err; /** Hardware specific return code */ odp_crypto_hw_err_t hw_err; -} odp_crypto_compl_status_t; +} odp_crypto_op_status_t; + +/** @deprecated Use ODP_DEPRECATE(odp_crypto_op_status_t) instead */ +typedef odp_crypto_op_status_t ODP_DEPRECATE(odp_crypto_compl_status_t); /** * Crypto API operation result @@ -460,27 +504,51 @@ typedef struct odp_crypto_op_result { /** Request completed successfully */ odp_bool_t ok; - /** User context from request */ - void *ctx; + /** User context from request + * + * @deprecated No need to pass context around sync calls + * */ + void *ODP_DEPRECATE(ctx); /** Output packet */ odp_packet_t pkt; /** Cipher status */ - odp_crypto_compl_status_t cipher_status; + odp_crypto_op_status_t cipher_status; /** Authentication status */ - odp_crypto_compl_status_t auth_status; + odp_crypto_op_status_t auth_status; } odp_crypto_op_result_t; /** + * Crypto packet API operation result + */ +typedef struct odp_crypto_packet_result_t { + /** Request completed successfully */ + odp_bool_t ok; + + /** Cipher status */ + odp_crypto_op_status_t cipher_status; + + /** Authentication status */ + odp_crypto_op_status_t auth_status; + +} odp_crypto_packet_result_t; + +/** * Crypto capabilities */ typedef struct odp_crypto_capability_t { /** Maximum number of crypto sessions */ uint32_t max_sessions; + /** Supported packet operation in SYNC mode */ + odp_support_t packet_sync_mode; + + /** Supported packet operation in ASYNC mode */ + odp_support_t packet_async_mode; + /** Supported cipher algorithms */ odp_crypto_cipher_algos_t ciphers; @@ -713,6 +781,94 @@ uint64_t ODP_DEPRECATE(odp_crypto_compl_to_u64)( void odp_crypto_session_param_init(odp_crypto_session_param_t *param); /** + * Return crypto processed packet that is associated with event + * + * Get packet handle to an crypto processed packet event. Event subtype must be + * ODP_EVENT_PACKET_CRYPTO. Crypto operation results can be examined with + * odp_crypto_packet_result(). + * + * Note: any invalid parameters will cause undefined behavior and may cause + * the application to abort or crash. + * + * @param ev Event handle + * + * @return Packet handle + */ +odp_packet_t odp_crypto_packet_from_event(odp_event_t ev); + +/** + * Convert crypto packet handle to event + * + * The packet handle must be an output of an crypto operation. + * + * @param pkt Packet handle from crypto operation + * + * @return Event handle + */ +odp_event_t odp_crypto_packet_to_event(odp_packet_t pkt); + +/** + * Get crypto operation results from an crypto processed packet + * + * Successful crypto operations of all types (SYNC and ASYNC) produce packets + * which contain crypto result metadata. This function copies the operation + * results from an crypto processed packet. Event subtype of this kind of + * packet is ODP_EVENT_PACKET_crypto. Results are undefined if a non-crypto + * processed packet is passed as input. + * + * @param packet An crypto processed packet (ODP_EVENT_PACKET_CRYPTO) + * @param[out] result Pointer to operation result for output + * + * @retval 0 On success + * @retval <0 On failure + */ +int odp_crypto_packet_result(odp_crypto_packet_result_t *result, + odp_packet_t packet); + +/** + * Crypto packet operation + * + * Performs the SYNC cryptographic operations specified during session creation + * on the packets. Caller should initialize pkt_out either with desired output + * packet handles or with ODP_PACKET_INVALID to make ODP allocate new packets + * from provided pool. All arrays should be of num_pkt size. + * + * @param pkt_in Packets to be processed + * @param[in,out] pkt_out Packet handle array specifyint resulting packets + * @param param Operation parameters array + * @param num_pkt Number of packets to be processed + * + * @return Number of input packets consumed (0 ... num_pkt) + * @retval <0 on failure + */ +int odp_crypto_packet_op(const odp_packet_t pkt_in[], + odp_packet_t pkt_out[], + const odp_crypto_packet_op_param_t param[], + int num_pkt); + +/** + * Crypto packet operation + * + * Performs the ASYNC cryptographic operations specified during session creation + * on the packets. Caller should initialize pkt_out either with desired output + * packet handles or with ODP_PACKET_INVALID to make ODP allocate new packets + * from provided pool. All arrays should be of num_pkt size. Resulting packets + * are returned through events. + * + * @param pkt_in Packets to be processed + * @param pkt_out Packet handle array specifying resulting packets + * @param param Operation parameters array + * @param num_pkt Number of packets to be processed + * + * @return Number of input packets consumed (0 ... num_pkt) + * @retval <0 on failure + */ +int odp_crypto_packet_op_enq(const odp_packet_t pkt_in[], + const odp_packet_t pkt_out[], + const odp_crypto_packet_op_param_t param[], + int num_pkt); + +/** * @} */ diff --git a/include/odp/arch/default/api/abi/event.h b/include/odp/arch/default/api/abi/event.h index 23b16efd..871d2cf9 100644 --- a/include/odp/arch/default/api/abi/event.h +++ b/include/odp/arch/default/api/abi/event.h @@ -37,7 +37,8 @@ typedef enum odp_event_type_t { typedef enum odp_event_subtype_t { ODP_EVENT_NO_SUBTYPE = 0, ODP_EVENT_PACKET_BASIC = 1, - ODP_EVENT_PACKET_IPSEC = 2 + ODP_EVENT_PACKET_CRYPTO = 2, + ODP_EVENT_PACKET_IPSEC = 3 } odp_event_subtype_t; /** diff --git a/platform/linux-generic/include/odp/api/plat/event_types.h b/platform/linux-generic/include/odp/api/plat/event_types.h index 1987bc8b..58ce0a8b 100644 --- a/platform/linux-generic/include/odp/api/plat/event_types.h +++ b/platform/linux-generic/include/odp/api/plat/event_types.h @@ -46,7 +46,8 @@ typedef enum odp_event_type_t { typedef enum odp_event_subtype_t { ODP_EVENT_NO_SUBTYPE = 0, ODP_EVENT_PACKET_BASIC = 1, - ODP_EVENT_PACKET_IPSEC = 2 + ODP_EVENT_PACKET_CRYPTO = 2, + ODP_EVENT_PACKET_IPSEC = 3 } odp_event_subtype_t; /** commit c6e3edd5dc52764e4f1772341dc568a549d7eaa9 Author: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Date: Thu Jun 29 18:05:57 2017 +0300 api: crypto: deprecate ASYNC mode and odp_crypto_compl_t Setting pref_mode to ASYNC now results in SYNC behaviour, so let's deprecate all ASYNC-related crypto items: pref_mode, odp_crypto_compl_t and related functions. Passing context through param/result is also deprecated, as odp_crypto_operation now works only in sync mode, thus there is no need to pass additional data around it. Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Petri Savolainen <petri.savolainen(a)nokia.com> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/example/ipsec/odp_ipsec.c b/example/ipsec/odp_ipsec.c index 9ed3aa20..1ed03585 100644 --- a/example/ipsec/odp_ipsec.c +++ b/example/ipsec/odp_ipsec.c @@ -393,9 +393,7 @@ void ipsec_init_post(crypto_api_mode_e api_mode) auth_sa, tun, api_mode, - entry->input, - completionq, - out_pool)) { + entry->input)) { EXAMPLE_ERR("Error: IPSec cache entry failed.\n" ); exit(EXIT_FAILURE); @@ -662,7 +660,6 @@ pkt_disposition_e do_ipsec_in_classify(odp_packet_t pkt, /* Initialize parameters block */ memset(&params, 0, sizeof(params)); - params.ctx = ctx; params.session = entry->state.session; params.pkt = pkt; params.out_pkt = entry->in_place ? pkt : ODP_PACKET_INVALID; @@ -843,7 +840,6 @@ pkt_disposition_e do_ipsec_out_classify(odp_packet_t pkt, /* Initialize parameters block */ memset(&params, 0, sizeof(params)); params.session = entry->state.session; - params.ctx = ctx; params.pkt = pkt; params.out_pkt = entry->in_place ? pkt : ODP_PACKET_INVALID; @@ -1085,14 +1081,6 @@ int pktio_thread(void *arg EXAMPLE_UNUSED) } ctx->state = PKT_STATE_INPUT_VERIFY; } - } else if (ODP_EVENT_CRYPTO_COMPL == odp_event_type(ev)) { - odp_crypto_compl_t compl; - - compl = odp_crypto_compl_from_event(ev); - odp_crypto_compl_result(compl, &result); - odp_crypto_compl_free(compl); - pkt = result.pkt; - ctx = result.ctx; } else { abort(); } diff --git a/example/ipsec/odp_ipsec_cache.c b/example/ipsec/odp_ipsec_cache.c index dba0ea0a..e4150336 100644 --- a/example/ipsec/odp_ipsec_cache.c +++ b/example/ipsec/odp_ipsec_cache.c @@ -40,9 +40,7 @@ int create_ipsec_cache_entry(sa_db_entry_t *cipher_sa, sa_db_entry_t *auth_sa, tun_db_entry_t *tun, crypto_api_mode_e api_mode, - odp_bool_t in, - odp_queue_t completionq, - odp_pool_t out_pool) + odp_bool_t in) { odp_crypto_session_param_t params; ipsec_cache_entry_t *entry; @@ -65,15 +63,8 @@ int create_ipsec_cache_entry(sa_db_entry_t *cipher_sa, /* Setup parameters and call crypto library to create session */ params.op = (in) ? ODP_CRYPTO_OP_DECODE : ODP_CRYPTO_OP_ENCODE; params.auth_cipher_text = TRUE; - if (CRYPTO_API_SYNC == api_mode) { - params.pref_mode = ODP_CRYPTO_SYNC; - params.compl_queue = ODP_QUEUE_INVALID; - params.output_pool = ODP_POOL_INVALID; - } else { - params.pref_mode = ODP_CRYPTO_ASYNC; - params.compl_queue = completionq; - params.output_pool = out_pool; - } + params.compl_queue = ODP_QUEUE_INVALID; + params.output_pool = ODP_POOL_INVALID; if (CRYPTO_API_ASYNC_NEW_BUFFER == api_mode) entry->in_place = FALSE; diff --git a/example/ipsec/odp_ipsec_cache.h b/example/ipsec/odp_ipsec_cache.h index 7a4b95cd..ce37ccce 100644 --- a/example/ipsec/odp_ipsec_cache.h +++ b/example/ipsec/odp_ipsec_cache.h @@ -85,8 +85,6 @@ void init_ipsec_cache(void); * @param tun Tunnel DB entry pointer * @param api_mode Crypto API mode for testing * @param in Direction (input versus output) - * @param completionq Completion queue - * @param out_pool Output buffer pool * * @return 0 if successful else -1 */ @@ -94,9 +92,7 @@ int create_ipsec_cache_entry(sa_db_entry_t *cipher_sa, sa_db_entry_t *auth_sa, tun_db_entry_t *tun, crypto_api_mode_e api_mode, - odp_bool_t in, - odp_queue_t completionq, - odp_pool_t out_pool); + odp_bool_t in); /** * Find a matching IPsec cache entry for input packet diff --git a/include/odp/api/spec/crypto.h b/include/odp/api/spec/crypto.h index 470cba05..b9d42937 100644 --- a/include/odp/api/spec/crypto.h +++ b/include/odp/api/spec/crypto.h @@ -270,8 +270,11 @@ typedef struct odp_crypto_session_param_t { */ odp_bool_t auth_cipher_text; - /** Preferred sync vs. async */ - odp_crypto_op_mode_t pref_mode; + /** Preferred sync vs. async + * + * @deprecated no-op now, odp_crypto_operation() will always process + * data in non-posted mode */ + odp_crypto_op_mode_t ODP_DEPRECATE(pref_mode); /** Cipher algorithm * @@ -334,8 +337,11 @@ typedef struct odp_crypto_op_param_t { /** Session handle from creation */ odp_crypto_session_t session; - /** User context */ - void *ctx; + /** User context + * + * @deprecated No need to pass context around sync calls + * */ + void *ODP_DEPRECATE(ctx); /** Input packet * @@ -618,7 +624,8 @@ int odp_crypto_session_destroy(odp_crypto_session_t session); * * @return crypto completion handle */ -odp_crypto_compl_t odp_crypto_compl_from_event(odp_event_t ev); +ODP_DEPRECATE(odp_crypto_compl_t) ODP_DEPRECATE(odp_crypto_compl_from_event)( + odp_event_t ev); /** * Convert crypto completion handle to event handle @@ -627,14 +634,16 @@ odp_crypto_compl_t odp_crypto_compl_from_event(odp_event_t ev); * * @return Event handle */ -odp_event_t odp_crypto_compl_to_event(odp_crypto_compl_t completion_event); +odp_event_t ODP_DEPRECATE(odp_crypto_compl_to_event)( + ODP_DEPRECATE(odp_crypto_compl_t) completion_event); /** * Release crypto completion event * * @param completion_event Completion event we are done accessing */ -void odp_crypto_compl_free(odp_crypto_compl_t completion_event); +void ODP_DEPRECATE(odp_crypto_compl_free)( + ODP_DEPRECATE(odp_crypto_compl_t) completion_event); /** * Crypto per packet operation @@ -662,8 +671,9 @@ int odp_crypto_operation(odp_crypto_op_param_t *param, * @param completion_event Event containing operation results * @param result Pointer to result structure */ -void odp_crypto_compl_result(odp_crypto_compl_t completion_event, - odp_crypto_op_result_t *result); +void ODP_DEPRECATE(odp_crypto_compl_result)( + ODP_DEPRECATE(odp_crypto_compl_t) completion_event, + odp_crypto_op_result_t *result); /** * Get printable value for an odp_crypto_session_t @@ -689,7 +699,8 @@ uint64_t odp_crypto_session_to_u64(odp_crypto_session_t hdl); * to enable applications to generate a printable value that represents * an odp_crypto_compl_t handle. */ -uint64_t odp_crypto_compl_to_u64(odp_crypto_compl_t hdl); +uint64_t ODP_DEPRECATE(odp_crypto_compl_to_u64)( + ODP_DEPRECATE(odp_crypto_compl_t) hdl); /** * Initialize crypto session parameters diff --git a/include/odp/arch/default/api/abi/crypto.h b/include/odp/arch/default/api/abi/crypto.h index f0793a19..2d02eee6 100644 --- a/include/odp/arch/default/api/abi/crypto.h +++ b/include/odp/arch/default/api/abi/crypto.h @@ -11,6 +11,8 @@ extern "C" { #endif +#include <odp/api/deprecated.h> + #include <stdint.h> /** @internal Dummy type for strong typing */ @@ -23,7 +25,7 @@ typedef struct { char dummy; /**< @internal Dummy */ } _odp_abi_crypto_compl_t; #define ODP_CRYPTO_SESSION_INVALID (0xffffffffffffffffULL) typedef uint64_t odp_crypto_session_t; -typedef _odp_abi_crypto_compl_t *odp_crypto_compl_t; +typedef _odp_abi_crypto_compl_t *ODP_DEPRECATE(odp_crypto_compl_t); /** * @} diff --git a/include/odp/arch/default/api/abi/event.h b/include/odp/arch/default/api/abi/event.h index ab3c0f75..23b16efd 100644 --- a/include/odp/arch/default/api/abi/event.h +++ b/include/odp/arch/default/api/abi/event.h @@ -11,6 +11,8 @@ extern "C" { #endif +#include <odp/api/deprecated.h> + #include <stdint.h> /** @internal Dummy type for strong typing */ @@ -28,7 +30,7 @@ typedef enum odp_event_type_t { ODP_EVENT_BUFFER = 1, ODP_EVENT_PACKET = 2, ODP_EVENT_TIMEOUT = 3, - ODP_EVENT_CRYPTO_COMPL = 4, + ODP_DEPRECATE(ODP_EVENT_CRYPTO_COMPL) = 4, ODP_EVENT_IPSEC_STATUS = 5 } odp_event_type_t; diff --git a/platform/linux-generic/include/odp/api/plat/crypto_types.h b/platform/linux-generic/include/odp/api/plat/crypto_types.h index 2cc747eb..23e1cc42 100644 --- a/platform/linux-generic/include/odp/api/plat/crypto_types.h +++ b/platform/linux-generic/include/odp/api/plat/crypto_types.h @@ -22,6 +22,7 @@ extern "C" { #if ODP_ABI_COMPAT == 1 #include <odp/api/abi/crypto.h> #else +#include <odp/api/deprecated.h> /** @ingroup odp_crypto * @{ @@ -30,7 +31,7 @@ extern "C" { #define ODP_CRYPTO_SESSION_INVALID (0xffffffffffffffffULL) typedef uint64_t odp_crypto_session_t; -typedef ODP_HANDLE_T(odp_crypto_compl_t); +typedef ODP_HANDLE_T(ODP_DEPRECATE(odp_crypto_compl_t)); /** * @} diff --git a/platform/linux-generic/include/odp/api/plat/event_types.h b/platform/linux-generic/include/odp/api/plat/event_types.h index 5b3a07e3..1987bc8b 100644 --- a/platform/linux-generic/include/odp/api/plat/event_types.h +++ b/platform/linux-generic/include/odp/api/plat/event_types.h @@ -23,6 +23,7 @@ extern "C" { #include <odp/api/abi/event.h> #else +#include <odp/api/deprecated.h> #include <odp/api/std_types.h> #include <odp/api/plat/strong_types.h> @@ -38,7 +39,7 @@ typedef enum odp_event_type_t { ODP_EVENT_BUFFER = 1, ODP_EVENT_PACKET = 2, ODP_EVENT_TIMEOUT = 3, - ODP_EVENT_CRYPTO_COMPL = 4, + ODP_DEPRECATE(ODP_EVENT_CRYPTO_COMPL) = 4, ODP_EVENT_IPSEC_STATUS = 5 } odp_event_type_t; diff --git a/platform/linux-generic/odp_crypto.c b/platform/linux-generic/odp_crypto.c index b23b5b6b..075bb02b 100644 --- a/platform/linux-generic/odp_crypto.c +++ b/platform/linux-generic/odp_crypto.c @@ -885,7 +885,9 @@ odp_crypto_operation(odp_crypto_op_param_t *param, } /* Fill in result */ +#if ODP_DEPRECATED_API local_result.ctx = param->ctx; +#endif local_result.pkt = param->out_pkt; local_result.cipher_status.alg_err = rc_cipher; local_result.cipher_status.hw_err = ODP_CRYPTO_HW_ERR_NONE; @@ -1045,6 +1047,7 @@ int32_t odp_random_test_data(uint8_t *buf, uint32_t len, uint64_t *seed) return len; } +#if ODP_DEPRECATED_API odp_crypto_compl_t odp_crypto_compl_from_event(odp_event_t ev) { /* This check not mandated by the API specification */ @@ -1077,6 +1080,12 @@ odp_crypto_compl_free(odp_crypto_compl_t completion_event) odp_buffer_free(odp_buffer_from_event(ev)); } +uint64_t odp_crypto_compl_to_u64(odp_crypto_compl_t hdl) +{ + return _odp_pri(hdl); +} +#endif + void odp_crypto_session_param_init(odp_crypto_session_param_t *param) { memset(param, 0, sizeof(odp_crypto_session_param_t)); @@ -1086,8 +1095,3 @@ uint64_t odp_crypto_session_to_u64(odp_crypto_session_t hdl) { return (uint64_t)hdl; } - -uint64_t odp_crypto_compl_to_u64(odp_crypto_compl_t hdl) -{ - return _odp_pri(hdl); -} diff --git a/platform/linux-generic/odp_event.c b/platform/linux-generic/odp_event.c index 23b410d8..56cc90b0 100644 --- a/platform/linux-generic/odp_event.c +++ b/platform/linux-generic/odp_event.c @@ -46,9 +46,11 @@ void odp_event_free(odp_event_t event) case ODP_EVENT_TIMEOUT: odp_timeout_free(odp_timeout_from_event(event)); break; +#if ODP_DEPRECATED_API case ODP_EVENT_CRYPTO_COMPL: odp_crypto_compl_free(odp_crypto_compl_from_event(event)); break; +#endif default: ODP_ABORT("Invalid event type: %d\n", odp_event_type(event)); } diff --git a/test/common_plat/performance/odp_crypto.c b/test/common_plat/performance/odp_crypto.c index c3dd6d38..bc29a888 100644 --- a/test/common_plat/performance/odp_crypto.c +++ b/test/common_plat/performance/odp_crypto.c @@ -427,7 +427,6 @@ create_session_from_config(odp_crypto_session_t *session, odp_crypto_session_param_init(&params); memcpy(&params, &config->session, sizeof(odp_crypto_session_param_t)); params.op = ODP_CRYPTO_OP_ENCODE; - params.pref_mode = ODP_CRYPTO_SYNC; /* Lookup the packet pool */ pkt_pool = odp_pool_lookup("packet_pool"); @@ -571,7 +570,7 @@ run_measure_one(crypto_args_t *cargs, packets_sent++; } - if (!posted) { + if (1) { packets_received++; if (cargs->debug_packets) { mem = odp_packet_data(params.out_pkt); @@ -587,45 +586,6 @@ run_measure_one(crypto_args_t *cargs, odp_packet_free(params.out_pkt); } } - } else { - odp_event_t ev; - odp_crypto_compl_t compl; - odp_crypto_op_result_t result; - odp_packet_t out_pkt; - - if (cargs->schedule) - ev = odp_schedule(NULL, - ODP_SCHED_NO_WAIT); - else - ev = odp_queue_deq(out_queue); - - while (ev != ODP_EVENT_INVALID) { - compl = odp_crypto_compl_from_event(ev); - odp_crypto_compl_result(compl, &result); - odp_crypto_compl_free(compl); - out_pkt = result.pkt; - - if (cargs->debug_packets) { - mem = odp_packet_data(out_pkt); - print_mem("Receieved encrypted packet", - mem, - payload_length + - config-> - session.auth_digest_len); - } - if (cargs->reuse_packet) { - params.pkt = out_pkt; - params.out_pkt = ODP_PACKET_INVALID; - } else { - odp_packet_free(out_pkt); - } - packets_received++; - if (cargs->schedule) - ev = odp_schedule(NULL, - ODP_SCHED_NO_WAIT); - else - ev = odp_queue_deq(out_queue); - }; } } diff --git a/test/common_plat/validation/api/crypto/crypto.c b/test/common_plat/validation/api/crypto/crypto.c index 94beb2f1..fde95ab7 100644 --- a/test/common_plat/validation/api/crypto/crypto.c +++ b/test/common_plat/validation/api/crypto/crypto.c @@ -15,8 +15,6 @@ odp_suiteinfo_t crypto_suites[] = { {ODP_CRYPTO_SYNC_INP, crypto_suite_sync_init, crypto_suite_term, crypto_suite}, - {ODP_CRYPTO_ASYNC_INP, crypto_suite_async_init, crypto_suite_term, - crypto_suite}, ODP_SUITE_INFO_NULL, }; diff --git a/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c b/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c index 1d7d49a2..d42ecb68 100644 --- a/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c +++ b/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c @@ -14,7 +14,6 @@ #define MAX_ALG_CAPA 32 struct suite_context_s { - odp_crypto_op_mode_t pref_mode; odp_pool_t pool; odp_queue_t queue; }; @@ -92,8 +91,6 @@ static void alg_test(odp_crypto_op_t op, int rc; odp_crypto_ses_create_err_t status; odp_bool_t posted; - odp_event_t event; - odp_crypto_compl_t compl_event; odp_crypto_op_result_t result; odp_crypto_session_param_t ses_params; odp_crypto_op_param_t op_params; @@ -196,7 +193,6 @@ static void alg_test(odp_crypto_op_t op, odp_crypto_session_param_init(&ses_params); ses_params.op = op; ses_params.auth_cipher_text = false; - ses_params.pref_mode = suite_context.pref_mode; ses_params.cipher_alg = cipher_alg; ses_params.auth_alg = auth_alg; ses_params.compl_queue = suite_context.queue; @@ -225,7 +221,6 @@ static void alg_test(odp_crypto_op_t op, op_params.session = session; op_params.pkt = pkt; op_params.out_pkt = pkt; - op_params.ctx = (void *)0xdeadbeef; if (cipher_range) { op_params.cipher_range = *cipher_range; @@ -258,27 +253,8 @@ static void alg_test(odp_crypto_op_t op, goto cleanup; } - if (posted) { - /* Poll completion queue for results */ - do { - event = odp_queue_deq(suite_context.queue); - } while (event == ODP_EVENT_INVALID); - - CU_ASSERT(ODP_EVENT_CRYPTO_COMPL == odp_event_type(event)); - CU_ASSERT(ODP_EVENT_NO_SUBTYPE == odp_event_subtype(event)); - CU_ASSERT(ODP_EVENT_CRYPTO_COMPL == - odp_event_types(event, &subtype)); - CU_ASSERT(ODP_EVENT_NO_SUBTYPE == subtype); - - compl_event = odp_crypto_compl_from_event(event); - CU_ASSERT(odp_crypto_compl_to_u64(compl_event) == - odp_crypto_compl_to_u64(odp_crypto_compl_from_event(event))); - odp_crypto_compl_result(compl_event, &result); - odp_crypto_compl_free(compl_event); - } - + CU_ASSERT(posted == 0); CU_ASSERT(result.pkt == pkt); - CU_ASSERT(result.ctx == (void *)0xdeadbeef); CU_ASSERT(ODP_EVENT_PACKET == odp_event_type(odp_packet_to_event(result.pkt))); CU_ASSERT(ODP_EVENT_PACKET_BASIC == @@ -1519,20 +1495,6 @@ int crypto_suite_sync_init(void) return -1; suite_context.queue = ODP_QUEUE_INVALID; - suite_context.pref_mode = ODP_CRYPTO_SYNC; - return 0; -} - -int crypto_suite_async_init(void) -{ - suite_context.pool = odp_pool_lookup("packet_pool"); - if (suite_context.pool == ODP_POOL_INVALID) - return -1; - suite_context.queue = odp_queue_lookup("crypto-out"); - if (suite_context.queue == ODP_QUEUE_INVALID) - return -1; - - suite_context.pref_mode = ODP_CRYPTO_ASYNC; return 0; } diff --git a/test/common_plat/validation/api/crypto/odp_crypto_test_inp.h b/test/common_plat/validation/api/crypto/odp_crypto_test_inp.h index 0f693379..86ecce8e 100644 --- a/test/common_plat/validation/api/crypto/odp_crypto_test_inp.h +++ b/test/common_plat/validation/api/crypto/odp_crypto_test_inp.h @@ -9,14 +9,12 @@ #include <odp_cunit_common.h> /* Suite names */ -#define ODP_CRYPTO_ASYNC_INP "odp_crypto_async_inp" #define ODP_CRYPTO_SYNC_INP "odp_crypto_sync_inp" /* Suite test array */ extern odp_testinfo_t crypto_suite[]; int crypto_suite_sync_init(void); -int crypto_suite_async_init(void); int crypto_suite_term(void); #endif commit 26644862f90f4dc2f6ed99534d16f8b4d136de82 Author: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Date: Sat Jul 1 12:11:50 2017 +0300 linux-gen: fix ODP_HANDLE_T expansion with c++ if type is compound Currently expansion of ODP_HANDLE_T defined in strong_types.h would result in concatenating undercrore with 'type'. If type defined using macro, this can result in underscore being concatenated with macro name, resulting in further errors. Use two-stage ODP_HANDLE_T expansion in C++ case so that macro params are expanded first, before being concatenated with underscore. Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Petri Savolainen <petri.savolainen(a)nokia.com> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp/api/plat/strong_types.h b/platform/linux-generic/include/odp/api/plat/strong_types.h index a53d7635..2e3070f8 100644 --- a/platform/linux-generic/include/odp/api/plat/strong_types.h +++ b/platform/linux-generic/include/odp/api/plat/strong_types.h @@ -17,7 +17,9 @@ /** Use strong typing for ODP types */ #ifdef __cplusplus -#define ODP_HANDLE_T(type) struct _##type { uint8_t unused_dummy_var; } *type +/* Allow type to be expanded before concatenation with underscore */ +#define _ODP_HANDLE_T(type) struct _##type { uint8_t unused_dummy_var; } *type +#define ODP_HANDLE_T(type) _ODP_HANDLE_T(type) #else #define odp_handle_t struct { uint8_t unused_dummy_var; } * /** C/C++ helper macro for strong typing */ commit e777674f2d484e98e3329c3e22022fb4c6d86069 Author: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Date: Tue Jul 11 15:58:16 2017 +0300 linux-gen: drop crypto async mode implementation Currently even if application asks odp_crypto to use ASYNC mode via pref_mode, implemetation can still return packets syncrhonously. In preparation of forthcoming changes, make linux-generic implementation always return packets in sync way. Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer(a)linaro.org> Reviewed-by: Petri Savolainen <petri.savolainen(a)nokia.com> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_crypto_internal.h b/platform/linux-generic/include/odp_crypto_internal.h index c3b70b23..52b94d07 100644 --- a/platform/linux-generic/include/odp_crypto_internal.h +++ b/platform/linux-generic/include/odp_crypto_internal.h @@ -56,14 +56,6 @@ struct odp_crypto_generic_session { }; /** - * Per packet operation result - */ -typedef struct odp_crypto_generic_op_result { - uint32_t magic; - odp_crypto_op_result_t result; -} odp_crypto_generic_op_result_t; - -/** * Per session creation operation result */ typedef struct odp_crypto_generic_session_result { diff --git a/platform/linux-generic/include/odp_packet_internal.h b/platform/linux-generic/include/odp_packet_internal.h index d513f05e..b74e0cb1 100644 --- a/platform/linux-generic/include/odp_packet_internal.h +++ b/platform/linux-generic/include/odp_packet_internal.h @@ -129,9 +129,6 @@ typedef struct { /* Classifier destination queue */ queue_t dst_queue; - /* Result for crypto */ - odp_crypto_generic_op_result_t op_result; - /* Packet data storage */ uint8_t data[0]; } odp_packet_hdr_t; @@ -172,7 +169,6 @@ static inline void copy_packet_cls_metadata(odp_packet_hdr_t *src_hdr, dst_hdr->dst_queue = src_hdr->dst_queue; dst_hdr->flow_hash = src_hdr->flow_hash; dst_hdr->timestamp = src_hdr->timestamp; - dst_hdr->op_result = src_hdr->op_result; } static inline void pull_tail(odp_packet_hdr_t *pkt_hdr, uint32_t len) diff --git a/platform/linux-generic/odp_crypto.c b/platform/linux-generic/odp_crypto.c index 0e2adad1..b23b5b6b 100644 --- a/platform/linux-generic/odp_crypto.c +++ b/platform/linux-generic/odp_crypto.c @@ -88,14 +88,6 @@ struct odp_crypto_global_s { static odp_crypto_global_t *global; static -odp_crypto_generic_op_result_t *get_op_result_from_event(odp_event_t ev) -{ - odp_packet_hdr_t *hdr = odp_packet_hdr(odp_packet_from_event(ev)); - - return &hdr->op_result; -} - -static odp_crypto_generic_session_t *alloc_session(void) { odp_crypto_generic_session_t *session = NULL; @@ -903,37 +895,14 @@ odp_crypto_operation(odp_crypto_op_param_t *param, (rc_cipher == ODP_CRYPTO_ALG_ERR_NONE) && (rc_auth == ODP_CRYPTO_ALG_ERR_NONE); - /* If specified during creation post event to completion queue */ - if (ODP_QUEUE_INVALID != session->p.compl_queue) { - odp_event_t completion_event; - odp_crypto_generic_op_result_t *op_result; - odp_buffer_t buf; - - /* Linux generic will always use packet for completion event */ - completion_event = odp_packet_to_event(param->out_pkt); - buf = odp_buffer_from_event(completion_event); - _odp_buffer_event_type_set(buf, ODP_EVENT_CRYPTO_COMPL); - _odp_buffer_event_subtype_set(buf, ODP_EVENT_NO_SUBTYPE); - /* Asynchronous, build result (no HW so no errors) and send it*/ - op_result = get_op_result_from_event(completion_event); - op_result->magic = OP_RESULT_MAGIC; - op_result->result = local_result; - if (odp_queue_enq(session->p.compl_queue, completion_event)) { - odp_event_free(completion_event); - goto err; - } + /* Synchronous, simply return results */ + if (!result) + goto err; + *result = local_result; - /* Indicate to caller operation was async */ - *posted = 1; - } else { - /* Synchronous, simply return results */ - if (!result) - goto err; - *result = local_result; + /* Indicate to caller operation was sync */ + *posted = 0; - /* Indicate to caller operation was sync */ - *posted = 0; - } return 0; err: @@ -1093,25 +1062,19 @@ void odp_crypto_compl_result(odp_crypto_compl_t completion_event, odp_crypto_op_result_t *result) { - odp_event_t ev = odp_crypto_compl_to_event(completion_event); - odp_crypto_generic_op_result_t *op_result; + (void)completion_event; + (void)result; - op_result = get_op_result_from_event(ev); - - if (OP_RESULT_MAGIC != op_result->magic) - ODP_ABORT(); - - memcpy(result, &op_result->result, sizeof(*result)); + /* We won't get such events anyway, so there can be no result */ + ODP_ASSERT(0); } void odp_crypto_compl_free(odp_crypto_compl_t completion_event) { - odp_buffer_t buf = - odp_buffer_from_event((odp_event_t)completion_event); + odp_event_t ev = odp_crypto_compl_to_event(completion_event); - _odp_buffer_event_type_set(buf, ODP_EVENT_PACKET); - _odp_buffer_event_subtype_set(buf, ODP_EVENT_PACKET_BASIC); + odp_buffer_free(odp_buffer_from_event(ev)); } void odp_crypto_session_param_init(odp_crypto_session_param_t *param) diff --git a/platform/linux-generic/odp_packet.c b/platform/linux-generic/odp_packet.c index 3789feca..22358438 100644 --- a/platform/linux-generic/odp_packet.c +++ b/platform/linux-generic/odp_packet.c @@ -166,7 +166,6 @@ static inline void packet_seg_copy_md(odp_packet_hdr_t *dst, dst->dst_queue = src->dst_queue; dst->flow_hash = src->flow_hash; dst->timestamp = src->timestamp; - dst->op_result = src->op_result; /* buffer header side packet metadata */ dst->buf_hdr.buf_u64 = src->buf_hdr.buf_u64; ----------------------------------------------------------------------- Summary of changes: example/ipsec/odp_ipsec.c | 124 ++++--- example/ipsec/odp_ipsec_cache.c | 6 +- example/ipsec/odp_ipsec_cache.h | 1 + example/ipsec/odp_ipsec_misc.h | 2 +- include/odp/api/spec/crypto.h | 207 +++++++++-- include/odp/arch/default/api/abi/crypto.h | 4 +- include/odp/arch/default/api/abi/event.h | 7 +- .../include/odp/api/plat/crypto_types.h | 3 +- .../include/odp/api/plat/event_types.h | 6 +- .../include/odp/api/plat/strong_types.h | 4 +- .../linux-generic/include/odp_crypto_internal.h | 11 +- .../linux-generic/include/odp_packet_internal.h | 5 +- platform/linux-generic/odp_crypto.c | 386 ++++++++++++++------- platform/linux-generic/odp_event.c | 2 + platform/linux-generic/odp_packet.c | 1 - test/common_plat/performance/odp_crypto.c | 157 +++++---- test/common_plat/validation/api/crypto/crypto.c | 10 +- .../validation/api/crypto/odp_crypto_test_inp.c | 327 +++++++++++++---- .../validation/api/crypto/odp_crypto_test_inp.h | 8 +- 19 files changed, 874 insertions(+), 397 deletions(-) hooks/post-receive --

9 years, 1 month

[COMMIT] branch master updated. v1.15.0.0-46-gd9473fe6

by git＠git-us.linaro.org

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via d9473fe65b531bfbe54d83d4e429d72e553c17dc (commit) via 98b09c8289e0888ce09f68f454a066a415597ea2 (commit) via f72ae9c821eed99323a98a012659acd655095bb3 (commit) via 08863bb877feb8643d1583e29912cc08f2ca8105 (commit) via d55156b2eacf07e88cbff9f51d60238536dfb7ad (commit) via 1239cc0c12ad2be11d329eebbe4df7fad05b03d7 (commit) via e3fce2fca6067b03f7fd18c9a438d2058521fe11 (commit) via 29ad363b5ab3c5e6569f20ddf63e64cc53eeb019 (commit) via 8ef745b7b78b49fc58d14e7b79b450c0bb560558 (commit) via d9c85310cb82f83010c839b6a1bbb5a1fa672ceb (commit) via 5a38050f431d904b7354cd50611304da0a601727 (commit) via 200782c3da2104a020f278691b3b23b8d55893e0 (commit) from 29641a1e0d0868c09f35f9f32e6c8491efb62b2d (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit d9473fe65b531bfbe54d83d4e429d72e553c17dc Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Tue Jul 4 12:53:43 2017 +0300 linux-gen: queue: remove type set from interface TM should not modify queue type. Remove type_set queue interface function. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_queue_if.h b/platform/linux-generic/include/odp_queue_if.h index 7ba5c3c5..410c6b79 100644 --- a/platform/linux-generic/include/odp_queue_if.h +++ b/platform/linux-generic/include/odp_queue_if.h @@ -71,7 +71,6 @@ typedef void (*queue_set_enq_deq_fn_t)(queue_t q_int, queue_enq_multi_fn_t enq_multi, queue_deq_fn_t deq, queue_deq_multi_fn_t deq_multi); -typedef void (*queue_set_type_fn_t)(queue_t q_int, odp_queue_type_t type); /* Queue functions towards other internal components */ typedef struct { @@ -90,7 +89,6 @@ typedef struct { queue_get_pktin_fn_t get_pktin; queue_set_pktin_fn_t set_pktin; queue_set_enq_deq_fn_t set_enq_deq_fn; - queue_set_type_fn_t set_type; } queue_fn_t; extern const queue_fn_t *queue_fn; diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c index ec225d00..d52814b2 100644 --- a/platform/linux-generic/odp_queue.c +++ b/platform/linux-generic/odp_queue.c @@ -769,11 +769,6 @@ static void queue_set_enq_deq_func(queue_t q_int, qentry->s.dequeue_multi = deq_multi; } -static void queue_set_type(queue_t q_int, odp_queue_type_t type) -{ - qentry_from_int(q_int)->s.type = type; -} - static queue_t queue_from_ext(odp_queue_t handle) { return qentry_to_int(handle_to_qentry(handle)); @@ -822,6 +817,5 @@ queue_fn_t queue_default_fn = { .set_pktout = queue_set_pktout, .get_pktin = queue_get_pktin, .set_pktin = queue_set_pktin, - .set_enq_deq_fn = queue_set_enq_deq_func, - .set_type = queue_set_type + .set_enq_deq_fn = queue_set_enq_deq_func }; diff --git a/platform/linux-generic/odp_traffic_mngr.c b/platform/linux-generic/odp_traffic_mngr.c index 1f254d75..ec2d331f 100644 --- a/platform/linux-generic/odp_traffic_mngr.c +++ b/platform/linux-generic/odp_traffic_mngr.c @@ -37,9 +37,6 @@ static const pkt_desc_t EMPTY_PKT_DESC = { .word = 0 }; #define MAX_PRIORITIES ODP_TM_MAX_PRIORITIES #define NUM_SHAPER_COLORS ODP_NUM_SHAPER_COLORS -/* Traffic manager queue */ -#define QUEUE_TYPE_TM 4 - static tm_prop_t basic_prop_tbl[MAX_PRIORITIES][NUM_SHAPER_COLORS] = { [0] = { [ODP_TM_SHAPER_GREEN] = { 0, DECR_BOTH }, @@ -3926,7 +3923,6 @@ odp_tm_queue_t odp_tm_queue_create(odp_tm_t odp_tm, return ODP_TM_INVALID; } tm_queue_obj->tm_qentry = queue_fn->from_ext(queue); - queue_fn->set_type(tm_queue_obj->tm_qentry, QUEUE_TYPE_TM); queue_fn->set_enq_deq_fn(tm_queue_obj->tm_qentry, queue_tm_reenq, queue_tm_reenq_multi, NULL, NULL); commit 98b09c8289e0888ce09f68f454a066a415597ea2 Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Tue Jul 4 12:53:42 2017 +0300 linux-gen: queue: combine queue interface set functions Decrease queue interface size by combining four queue operation set functions into a single function. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_queue_if.h b/platform/linux-generic/include/odp_queue_if.h index b5cbd515..7ba5c3c5 100644 --- a/platform/linux-generic/include/odp_queue_if.h +++ b/platform/linux-generic/include/odp_queue_if.h @@ -66,12 +66,11 @@ typedef void (*queue_set_pktout_fn_t)(queue_t q_int, odp_pktio_t pktio, typedef odp_pktin_queue_t (*queue_get_pktin_fn_t)(queue_t q_int); typedef void (*queue_set_pktin_fn_t)(queue_t q_int, odp_pktio_t pktio, int index); -typedef void (*queue_set_enq_fn_t)(queue_t q_int, queue_enq_fn_t func); -typedef void (*queue_set_enq_multi_fn_t)(queue_t q_int, - queue_enq_multi_fn_t func); -typedef void (*queue_set_deq_fn_t)(queue_t q_int, queue_deq_fn_t func); -typedef void (*queue_set_deq_multi_fn_t)(queue_t q_int, - queue_deq_multi_fn_t func); +typedef void (*queue_set_enq_deq_fn_t)(queue_t q_int, + queue_enq_fn_t enq, + queue_enq_multi_fn_t enq_multi, + queue_deq_fn_t deq, + queue_deq_multi_fn_t deq_multi); typedef void (*queue_set_type_fn_t)(queue_t q_int, odp_queue_type_t type); /* Queue functions towards other internal components */ @@ -90,10 +89,7 @@ typedef struct { queue_set_pktout_fn_t set_pktout; queue_get_pktin_fn_t get_pktin; queue_set_pktin_fn_t set_pktin; - queue_set_enq_fn_t set_enq_fn; - queue_set_enq_multi_fn_t set_enq_multi_fn; - queue_set_deq_fn_t set_deq_fn; - queue_set_deq_multi_fn_t set_deq_multi_fn; + queue_set_enq_deq_fn_t set_enq_deq_fn; queue_set_type_fn_t set_type; } queue_fn_t; diff --git a/platform/linux-generic/odp_packet_io.c b/platform/linux-generic/odp_packet_io.c index 0d825f91..6a181f53 100644 --- a/platform/linux-generic/odp_packet_io.c +++ b/platform/linux-generic/odp_packet_io.c @@ -1154,30 +1154,30 @@ int odp_pktio_stats_reset(odp_pktio_t pktio) return ret; } -static int pktin_enqueue(queue_t q_int ODP_UNUSED, - odp_buffer_hdr_t *buf_hdr ODP_UNUSED) +static int abort_pktin_enqueue(queue_t q_int ODP_UNUSED, + odp_buffer_hdr_t *buf_hdr ODP_UNUSED) { ODP_ABORT("attempted enqueue to a pktin queue"); return -1; } -static int pktin_enq_multi(queue_t q_int ODP_UNUSED, - odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, - int num ODP_UNUSED) +static int abort_pktin_enq_multi(queue_t q_int ODP_UNUSED, + odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, + int num ODP_UNUSED) { ODP_ABORT("attempted enqueue to a pktin queue"); return 0; } -static odp_buffer_hdr_t *pktout_dequeue(queue_t q_int ODP_UNUSED) +static odp_buffer_hdr_t *abort_pktout_dequeue(queue_t q_int ODP_UNUSED) { ODP_ABORT("attempted dequeue from a pktout queue"); return NULL; } -static int pktout_deq_multi(queue_t q_int ODP_UNUSED, - odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, - int num ODP_UNUSED) +static int abort_pktout_deq_multi(queue_t q_int ODP_UNUSED, + odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, + int num ODP_UNUSED) { ODP_ABORT("attempted dequeue from a pktout queue"); return 0; @@ -1272,12 +1272,11 @@ int odp_pktin_queue_config(odp_pktio_t pktio, if (mode == ODP_PKTIN_MODE_QUEUE) { queue_fn->set_pktin(q_int, pktio, i); - queue_fn->set_enq_fn(q_int, pktin_enqueue); - queue_fn->set_deq_fn(q_int, pktin_dequeue); - queue_fn->set_enq_multi_fn(q_int, - pktin_enq_multi); - queue_fn->set_deq_multi_fn(q_int, - pktin_deq_multi); + queue_fn->set_enq_deq_fn(q_int, + abort_pktin_enqueue, + abort_pktin_enq_multi, + pktin_dequeue, + pktin_deq_multi); } entry->s.in_queue[i].queue = queue; @@ -1401,10 +1400,11 @@ int odp_pktout_queue_config(odp_pktio_t pktio, queue_fn->set_pktout(q_int, pktio, i); /* Override default enqueue / dequeue functions */ - queue_fn->set_enq_fn(q_int, pktout_enqueue); - queue_fn->set_deq_fn(q_int, pktout_dequeue); - queue_fn->set_enq_multi_fn(q_int, pktout_enq_multi); - queue_fn->set_deq_multi_fn(q_int, pktout_deq_multi); + queue_fn->set_enq_deq_fn(q_int, + pktout_enqueue, + pktout_enq_multi, + abort_pktout_dequeue, + abort_pktout_deq_multi); entry->s.out_queue[i].queue = queue; } diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c index 60f88f09..ec225d00 100644 --- a/platform/linux-generic/odp_queue.c +++ b/platform/linux-generic/odp_queue.c @@ -748,24 +748,25 @@ static void queue_set_pktin(queue_t q_int, odp_pktio_t pktio, int index) qentry->s.pktin.index = index; } -static void queue_set_enq_func(queue_t q_int, queue_enq_fn_t func) +static void queue_set_enq_deq_func(queue_t q_int, + queue_enq_fn_t enq, + queue_enq_multi_fn_t enq_multi, + queue_deq_fn_t deq, + queue_deq_multi_fn_t deq_multi) { - qentry_from_int(q_int)->s.enqueue = func; -} + queue_entry_t *qentry = qentry_from_int(q_int); -static void queue_set_enq_multi_func(queue_t q_int, queue_enq_multi_fn_t func) -{ - qentry_from_int(q_int)->s.enqueue_multi = func; -} + if (enq) + qentry->s.enqueue = enq; -static void queue_set_deq_func(queue_t q_int, queue_deq_fn_t func) -{ - qentry_from_int(q_int)->s.dequeue = func; -} + if (enq_multi) + qentry->s.enqueue_multi = enq_multi; -static void queue_set_deq_multi_func(queue_t q_int, queue_deq_multi_fn_t func) -{ - qentry_from_int(q_int)->s.dequeue_multi = func; + if (deq) + qentry->s.dequeue = deq; + + if (deq_multi) + qentry->s.dequeue_multi = deq_multi; } static void queue_set_type(queue_t q_int, odp_queue_type_t type) @@ -821,9 +822,6 @@ queue_fn_t queue_default_fn = { .set_pktout = queue_set_pktout, .get_pktin = queue_get_pktin, .set_pktin = queue_set_pktin, - .set_enq_fn = queue_set_enq_func, - .set_enq_multi_fn = queue_set_enq_multi_func, - .set_deq_fn = queue_set_deq_func, - .set_deq_multi_fn = queue_set_deq_multi_func, + .set_enq_deq_fn = queue_set_enq_deq_func, .set_type = queue_set_type }; diff --git a/platform/linux-generic/odp_traffic_mngr.c b/platform/linux-generic/odp_traffic_mngr.c index 8b5f3187..1f254d75 100644 --- a/platform/linux-generic/odp_traffic_mngr.c +++ b/platform/linux-generic/odp_traffic_mngr.c @@ -3927,9 +3927,9 @@ odp_tm_queue_t odp_tm_queue_create(odp_tm_t odp_tm, } tm_queue_obj->tm_qentry = queue_fn->from_ext(queue); queue_fn->set_type(tm_queue_obj->tm_qentry, QUEUE_TYPE_TM); - queue_fn->set_enq_fn(tm_queue_obj->tm_qentry, queue_tm_reenq); - queue_fn->set_enq_multi_fn(tm_queue_obj->tm_qentry, - queue_tm_reenq_multi); + queue_fn->set_enq_deq_fn(tm_queue_obj->tm_qentry, + queue_tm_reenq, queue_tm_reenq_multi, + NULL, NULL); tm_system->queue_num_tbl[tm_queue_obj->queue_num - 1] = tm_queue_obj; odp_ticketlock_lock(&tm_system->tm_system_lock); commit f72ae9c821eed99323a98a012659acd655095bb3 Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Tue Jul 4 12:53:41 2017 +0300 linux-gen: pktio: remove unused header file odp_packet_io_queue.h is not needed anymore as the queue interface has replaced it. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index 45dd0f4d..33299564 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -145,7 +145,6 @@ noinst_HEADERS = \ ${srcdir}/include/odp_packet_internal.h \ ${srcdir}/include/odp_packet_io_internal.h \ ${srcdir}/include/odp_packet_io_ipc_internal.h \ - ${srcdir}/include/odp_packet_io_queue.h \ ${srcdir}/include/odp_packet_io_ring_internal.h \ ${srcdir}/include/odp_packet_netmap.h \ ${srcdir}/include/odp_packet_dpdk.h \ diff --git a/platform/linux-generic/include/odp_packet_io_queue.h b/platform/linux-generic/include/odp_packet_io_queue.h deleted file mode 100644 index 71333ddc..00000000 --- a/platform/linux-generic/include/odp_packet_io_queue.h +++ /dev/null @@ -1,48 +0,0 @@ -/* Copyright (c) 2013, Linaro Limited - * All rights reserved. - * - * SPDX-License-Identifier: BSD-3-Clause - */ - - -/** - * @file - * - * ODP packet IO - implementation internal - */ - -#ifndef ODP_PACKET_IO_QUEUE_H_ -#define ODP_PACKET_IO_QUEUE_H_ - -#ifdef __cplusplus -extern "C" { -#endif - -#include <odp_queue_if.h> -#include <odp_buffer_internal.h> -#include <odp_config_internal.h> - -/** Max nbr of pkts to receive in one burst (keep same as QUEUE_MULTI_MAX) */ -#define ODP_PKTIN_QUEUE_MAX_BURST CONFIG_BURST_SIZE -/* pktin_deq_multi() depends on the condition: */ -ODP_STATIC_ASSERT(ODP_PKTIN_QUEUE_MAX_BURST >= QUEUE_MULTI_MAX, - "ODP_PKTIN_DEQ_MULTI_MAX_ERROR"); - -int pktin_enqueue(queue_t queue, odp_buffer_hdr_t *buf_hdr); -odp_buffer_hdr_t *pktin_dequeue(queue_t queue); - -int pktin_enq_multi(queue_t queue, odp_buffer_hdr_t *buf_hdr[], int num); -int pktin_deq_multi(queue_t queue, odp_buffer_hdr_t *buf_hdr[], int num); - - -int pktout_enqueue(queue_t queue, odp_buffer_hdr_t *buf_hdr); -odp_buffer_hdr_t *pktout_dequeue(queue_t queue); - -int pktout_enq_multi(queue_t queue, odp_buffer_hdr_t *buf_hdr[], int num); -int pktout_deq_multi(queue_t queue, odp_buffer_hdr_t *buf_hdr[], int num); - -#ifdef __cplusplus -} -#endif - -#endif diff --git a/platform/linux-generic/odp_packet_io.c b/platform/linux-generic/odp_packet_io.c index 3ed51ac9..0d825f91 100644 --- a/platform/linux-generic/odp_packet_io.c +++ b/platform/linux-generic/odp_packet_io.c @@ -7,7 +7,6 @@ #include <odp/api/packet_io.h> #include <odp_packet_io_internal.h> -#include <odp_packet_io_queue.h> #include <odp/api/packet.h> #include <odp_packet_internal.h> #include <odp_internal.h> @@ -580,7 +579,7 @@ static inline int pktin_recv_buf(odp_pktin_queue_t queue, return num_rx; } -int pktout_enqueue(queue_t q_int, odp_buffer_hdr_t *buf_hdr) +static int pktout_enqueue(queue_t q_int, odp_buffer_hdr_t *buf_hdr) { odp_packet_t pkt = packet_from_buf_hdr(buf_hdr); int len = 1; @@ -593,13 +592,7 @@ int pktout_enqueue(queue_t q_int, odp_buffer_hdr_t *buf_hdr) return (nbr == len ? 0 : -1); } -odp_buffer_hdr_t *pktout_dequeue(queue_t q_int ODP_UNUSED) -{ - ODP_ABORT("attempted dequeue from a pktout queue"); - return NULL; -} - -int pktout_enq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int num) +static int pktout_enq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int num) { odp_packet_t pkt_tbl[QUEUE_MULTI_MAX]; int nbr; @@ -615,22 +608,7 @@ int pktout_enq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int num) return nbr; } -int pktout_deq_multi(queue_t q_int ODP_UNUSED, - odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, - int num ODP_UNUSED) -{ - ODP_ABORT("attempted dequeue from a pktout queue"); - return 0; -} - -int pktin_enqueue(queue_t q_int ODP_UNUSED, - odp_buffer_hdr_t *buf_hdr ODP_UNUSED) -{ - ODP_ABORT("attempted enqueue to a pktin queue"); - return -1; -} - -odp_buffer_hdr_t *pktin_dequeue(queue_t q_int) +static odp_buffer_hdr_t *pktin_dequeue(queue_t q_int) { odp_buffer_hdr_t *buf_hdr; odp_buffer_hdr_t *hdr_tbl[QUEUE_MULTI_MAX]; @@ -652,14 +630,7 @@ odp_buffer_hdr_t *pktin_dequeue(queue_t q_int) return buf_hdr; } -int pktin_enq_multi(queue_t q_int ODP_UNUSED, - odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, int num ODP_UNUSED) -{ - ODP_ABORT("attempted enqueue to a pktin queue"); - return 0; -} - -int pktin_deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int num) +static int pktin_deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int num) { int nbr; odp_buffer_hdr_t *hdr_tbl[QUEUE_MULTI_MAX]; @@ -1183,6 +1154,35 @@ int odp_pktio_stats_reset(odp_pktio_t pktio) return ret; } +static int pktin_enqueue(queue_t q_int ODP_UNUSED, + odp_buffer_hdr_t *buf_hdr ODP_UNUSED) +{ + ODP_ABORT("attempted enqueue to a pktin queue"); + return -1; +} + +static int pktin_enq_multi(queue_t q_int ODP_UNUSED, + odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, + int num ODP_UNUSED) +{ + ODP_ABORT("attempted enqueue to a pktin queue"); + return 0; +} + +static odp_buffer_hdr_t *pktout_dequeue(queue_t q_int ODP_UNUSED) +{ + ODP_ABORT("attempted dequeue from a pktout queue"); + return NULL; +} + +static int pktout_deq_multi(queue_t q_int ODP_UNUSED, + odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, + int num ODP_UNUSED) +{ + ODP_ABORT("attempted dequeue from a pktout queue"); + return 0; +} + int odp_pktin_queue_config(odp_pktio_t pktio, const odp_pktin_queue_param_t *param) { diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c index 4c85027b..60f88f09 100644 --- a/platform/linux-generic/odp_queue.c +++ b/platform/linux-generic/odp_queue.c @@ -19,7 +19,6 @@ #include <odp_schedule_if.h> #include <odp_config_internal.h> #include <odp_packet_io_internal.h> -#include <odp_packet_io_queue.h> #include <odp_debug_internal.h> #include <odp/api/hints.h> #include <odp/api/sync.h> commit 08863bb877feb8643d1583e29912cc08f2ca8105 Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Tue Jul 4 12:53:40 2017 +0300 linux-gen: pktio: convert queue handle only once Avoid queue handle conversion on fast path. Convert once after queue create and save value. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_packet_io_internal.h b/platform/linux-generic/include/odp_packet_io_internal.h index 89bb6f3a..93040681 100644 --- a/platform/linux-generic/include/odp_packet_io_internal.h +++ b/platform/linux-generic/include/odp_packet_io_internal.h @@ -24,6 +24,7 @@ extern "C" { #include <odp_align_internal.h> #include <odp_debug_internal.h> #include <odp_packet_io_ring_internal.h> +#include <odp_queue_if.h> #include <odp_config_internal.h> #include <odp/api/hints.h> @@ -165,6 +166,7 @@ struct pktio_entry { struct { odp_queue_t queue; + queue_t queue_int; odp_pktin_queue_t pktin; } in_queue[PKTIO_MAX_QUEUES]; diff --git a/platform/linux-generic/odp_packet_io.c b/platform/linux-generic/odp_packet_io.c index af1307c0..3ed51ac9 100644 --- a/platform/linux-generic/odp_packet_io.c +++ b/platform/linux-generic/odp_packet_io.c @@ -128,6 +128,7 @@ static void init_in_queues(pktio_entry_t *entry) for (i = 0; i < PKTIO_MAX_QUEUES; i++) { entry->s.in_queue[i].queue = ODP_QUEUE_INVALID; + entry->s.in_queue[i].queue_int = QUEUE_NULL; entry->s.in_queue[i].pktin = PKTIN_INVALID; } } @@ -305,6 +306,7 @@ static void destroy_in_queues(pktio_entry_t *entry, int num) if (entry->s.in_queue[i].queue != ODP_QUEUE_INVALID) { odp_queue_destroy(entry->s.in_queue[i].queue); entry->s.in_queue[i].queue = ODP_QUEUE_INVALID; + entry->s.in_queue[i].queue_int = QUEUE_NULL; } } } @@ -708,8 +710,7 @@ int sched_cb_pktin_poll(int pktio_index, int num_queue, int index[]) } for (idx = 0; idx < num_queue; idx++) { - queue_t qentry; - odp_queue_t queue; + queue_t q_int; odp_pktin_queue_t pktin = entry->s.in_queue[index[idx]].pktin; num = pktin_recv_buf(pktin, hdr_tbl, QUEUE_MULTI_MAX); @@ -722,9 +723,8 @@ int sched_cb_pktin_poll(int pktio_index, int num_queue, int index[]) return -1; } - queue = entry->s.in_queue[index[idx]].queue; - qentry = queue_fn->from_ext(queue); - queue_fn->enq_multi(qentry, hdr_tbl, num); + q_int = entry->s.in_queue[index[idx]].queue_int; + queue_fn->enq_multi(q_int, hdr_tbl, num); } return 0; @@ -1281,8 +1281,11 @@ int odp_pktin_queue_config(odp_pktio_t pktio, } entry->s.in_queue[i].queue = queue; + entry->s.in_queue[i].queue_int = q_int; + } else { entry->s.in_queue[i].queue = ODP_QUEUE_INVALID; + entry->s.in_queue[i].queue_int = QUEUE_NULL; } entry->s.in_queue[i].pktin.index = i; commit d55156b2eacf07e88cbff9f51d60238536dfb7ad Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Tue Jul 4 12:53:39 2017 +0300 linux-gen: pktio: dont use qentry for queue_t These are not any more queue_entry_t* but queue_t. Align variable names with the new type. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/odp_packet_io.c b/platform/linux-generic/odp_packet_io.c index 6ff35046..af1307c0 100644 --- a/platform/linux-generic/odp_packet_io.c +++ b/platform/linux-generic/odp_packet_io.c @@ -578,42 +578,42 @@ static inline int pktin_recv_buf(odp_pktin_queue_t queue, return num_rx; } -int pktout_enqueue(queue_t qentry, odp_buffer_hdr_t *buf_hdr) +int pktout_enqueue(queue_t q_int, odp_buffer_hdr_t *buf_hdr) { odp_packet_t pkt = packet_from_buf_hdr(buf_hdr); int len = 1; int nbr; - if (sched_fn->ord_enq_multi(qentry, (void **)buf_hdr, len, &nbr)) + if (sched_fn->ord_enq_multi(q_int, (void **)buf_hdr, len, &nbr)) return (nbr == len ? 0 : -1); - nbr = odp_pktout_send(queue_fn->get_pktout(qentry), &pkt, len); + nbr = odp_pktout_send(queue_fn->get_pktout(q_int), &pkt, len); return (nbr == len ? 0 : -1); } -odp_buffer_hdr_t *pktout_dequeue(queue_t qentry ODP_UNUSED) +odp_buffer_hdr_t *pktout_dequeue(queue_t q_int ODP_UNUSED) { ODP_ABORT("attempted dequeue from a pktout queue"); return NULL; } -int pktout_enq_multi(queue_t qentry, odp_buffer_hdr_t *buf_hdr[], int num) +int pktout_enq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int num) { odp_packet_t pkt_tbl[QUEUE_MULTI_MAX]; int nbr; int i; - if (sched_fn->ord_enq_multi(qentry, (void **)buf_hdr, num, &nbr)) + if (sched_fn->ord_enq_multi(q_int, (void **)buf_hdr, num, &nbr)) return nbr; for (i = 0; i < num; ++i) pkt_tbl[i] = packet_from_buf_hdr(buf_hdr[i]); - nbr = odp_pktout_send(queue_fn->get_pktout(qentry), pkt_tbl, num); + nbr = odp_pktout_send(queue_fn->get_pktout(q_int), pkt_tbl, num); return nbr; } -int pktout_deq_multi(queue_t qentry ODP_UNUSED, +int pktout_deq_multi(queue_t q_int ODP_UNUSED, odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, int num ODP_UNUSED) { @@ -621,49 +621,49 @@ int pktout_deq_multi(queue_t qentry ODP_UNUSED, return 0; } -int pktin_enqueue(queue_t qentry ODP_UNUSED, +int pktin_enqueue(queue_t q_int ODP_UNUSED, odp_buffer_hdr_t *buf_hdr ODP_UNUSED) { ODP_ABORT("attempted enqueue to a pktin queue"); return -1; } -odp_buffer_hdr_t *pktin_dequeue(queue_t qentry) +odp_buffer_hdr_t *pktin_dequeue(queue_t q_int) { odp_buffer_hdr_t *buf_hdr; odp_buffer_hdr_t *hdr_tbl[QUEUE_MULTI_MAX]; int pkts; - buf_hdr = queue_fn->deq(qentry); + buf_hdr = queue_fn->deq(q_int); if (buf_hdr != NULL) return buf_hdr; - pkts = pktin_recv_buf(queue_fn->get_pktin(qentry), + pkts = pktin_recv_buf(queue_fn->get_pktin(q_int), hdr_tbl, QUEUE_MULTI_MAX); if (pkts <= 0) return NULL; if (pkts > 1) - queue_fn->enq_multi(qentry, &hdr_tbl[1], pkts - 1); + queue_fn->enq_multi(q_int, &hdr_tbl[1], pkts - 1); buf_hdr = hdr_tbl[0]; return buf_hdr; } -int pktin_enq_multi(queue_t qentry ODP_UNUSED, +int pktin_enq_multi(queue_t q_int ODP_UNUSED, odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, int num ODP_UNUSED) { ODP_ABORT("attempted enqueue to a pktin queue"); return 0; } -int pktin_deq_multi(queue_t qentry, odp_buffer_hdr_t *buf_hdr[], int num) +int pktin_deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int num) { int nbr; odp_buffer_hdr_t *hdr_tbl[QUEUE_MULTI_MAX]; int pkts, i, j; - nbr = queue_fn->deq_multi(qentry, buf_hdr, num); + nbr = queue_fn->deq_multi(q_int, buf_hdr, num); if (odp_unlikely(nbr > num)) ODP_ABORT("queue_deq_multi req: %d, returned %d\n", num, nbr); @@ -673,7 +673,7 @@ int pktin_deq_multi(queue_t qentry, odp_buffer_hdr_t *buf_hdr[], int num) if (nbr == num) return nbr; - pkts = pktin_recv_buf(queue_fn->get_pktin(qentry), + pkts = pktin_recv_buf(queue_fn->get_pktin(q_int), hdr_tbl, QUEUE_MULTI_MAX); if (pkts <= 0) return nbr; @@ -686,7 +686,7 @@ int pktin_deq_multi(queue_t qentry, odp_buffer_hdr_t *buf_hdr[], int num) hdr_tbl[j] = hdr_tbl[i]; if (j) - queue_fn->enq_multi(qentry, hdr_tbl, j); + queue_fn->enq_multi(q_int, hdr_tbl, j); return nbr; } @@ -1193,6 +1193,7 @@ int odp_pktin_queue_config(odp_pktio_t pktio, unsigned i; int rc; odp_queue_t queue; + queue_t q_int; odp_pktin_queue_param_t default_param; if (param == NULL) { @@ -1267,16 +1268,15 @@ int odp_pktin_queue_config(odp_pktio_t pktio, return -1; } - if (mode == ODP_PKTIN_MODE_QUEUE) { - queue_t qentry; + q_int = queue_fn->from_ext(queue); - qentry = queue_fn->from_ext(queue); - queue_fn->set_pktin(qentry, pktio, i); - queue_fn->set_enq_fn(qentry, pktin_enqueue); - queue_fn->set_deq_fn(qentry, pktin_dequeue); - queue_fn->set_enq_multi_fn(qentry, + if (mode == ODP_PKTIN_MODE_QUEUE) { + queue_fn->set_pktin(q_int, pktio, i); + queue_fn->set_enq_fn(q_int, pktin_enqueue); + queue_fn->set_deq_fn(q_int, pktin_dequeue); + queue_fn->set_enq_multi_fn(q_int, pktin_enq_multi); - queue_fn->set_deq_multi_fn(qentry, + queue_fn->set_deq_multi_fn(q_int, pktin_deq_multi); } @@ -1374,7 +1374,7 @@ int odp_pktout_queue_config(odp_pktio_t pktio, for (i = 0; i < num_queues; i++) { odp_queue_t queue; odp_queue_param_t queue_param; - queue_t qentry; + queue_t q_int; char name[ODP_QUEUE_NAME_LEN]; int pktio_id = pktio_to_id(pktio); @@ -1394,14 +1394,14 @@ int odp_pktout_queue_config(odp_pktio_t pktio, return -1; } - qentry = queue_fn->from_ext(queue); - queue_fn->set_pktout(qentry, pktio, i); + q_int = queue_fn->from_ext(queue); + queue_fn->set_pktout(q_int, pktio, i); /* Override default enqueue / dequeue functions */ - queue_fn->set_enq_fn(qentry, pktout_enqueue); - queue_fn->set_deq_fn(qentry, pktout_dequeue); - queue_fn->set_enq_multi_fn(qentry, pktout_enq_multi); - queue_fn->set_deq_multi_fn(qentry, pktout_deq_multi); + queue_fn->set_enq_fn(q_int, pktout_enqueue); + queue_fn->set_deq_fn(q_int, pktout_dequeue); + queue_fn->set_enq_multi_fn(q_int, pktout_enq_multi); + queue_fn->set_deq_multi_fn(q_int, pktout_deq_multi); entry->s.out_queue[i].queue = queue; } commit 1239cc0c12ad2be11d329eebbe4df7fad05b03d7 Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Tue Jul 4 12:53:38 2017 +0300 linux-gen: cls: avoid queue conversions Use internal queue type instead of API handle. This way per packet queue conversions are avoided. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_packet_internal.h b/platform/linux-generic/include/odp_packet_internal.h index cdd139bb..d513f05e 100644 --- a/platform/linux-generic/include/odp_packet_internal.h +++ b/platform/linux-generic/include/odp_packet_internal.h @@ -27,6 +27,7 @@ extern "C" { #include <odp/api/crypto.h> #include <odp_crypto_internal.h> #include <odp/api/plat/packet_types.h> +#include <odp_queue_if.h> /** Minimum segment length expected by packet_parse_common() */ #define PACKET_PARSE_SEG_LEN 96 @@ -126,7 +127,7 @@ typedef struct { odp_time_t timestamp; /* Classifier destination queue */ - odp_queue_t dst_queue; + queue_t dst_queue; /* Result for crypto */ odp_crypto_generic_op_result_t op_result; diff --git a/platform/linux-generic/include/odp_queue_if.h b/platform/linux-generic/include/odp_queue_if.h index 168d0e9e..b5cbd515 100644 --- a/platform/linux-generic/include/odp_queue_if.h +++ b/platform/linux-generic/include/odp_queue_if.h @@ -48,6 +48,8 @@ typedef struct { typedef struct { char dummy; } _queue_t; typedef _queue_t *queue_t; +#define QUEUE_NULL ((queue_t)NULL) + typedef int (*queue_init_global_fn_t)(void); typedef int (*queue_term_global_fn_t)(void); typedef int (*queue_init_local_fn_t)(void); diff --git a/platform/linux-generic/odp_classification.c b/platform/linux-generic/odp_classification.c index 5f153e73..cc08b9f4 100644 --- a/platform/linux-generic/odp_classification.c +++ b/platform/linux-generic/odp_classification.c @@ -162,7 +162,7 @@ odp_cos_t odp_cls_cos_create(const char *name, odp_cls_cos_param_t *param) /* Packets are dropped if Queue or Pool is invalid*/ if (param->queue == ODP_QUEUE_INVALID) - queue = NULL; + queue = QUEUE_NULL; else queue = queue_fn->from_ext(param->queue); @@ -264,7 +264,7 @@ int odp_cos_queue_set(odp_cos_t cos_id, odp_queue_t queue_id) /* Locking is not required as intermittent stale data during CoS modification is acceptable*/ if (queue_id == ODP_QUEUE_INVALID) - cos->s.queue = NULL; + cos->s.queue = QUEUE_NULL; else cos->s.queue = queue_fn->from_ext(queue_id); return 0; @@ -279,7 +279,7 @@ odp_queue_t odp_cos_queue(odp_cos_t cos_id) return ODP_QUEUE_INVALID; } - if (!cos->s.queue) + if (cos->s.queue == QUEUE_NULL) return ODP_QUEUE_INVALID; return queue_fn->to_ext(cos->s.queue); @@ -841,12 +841,12 @@ int cls_classify_packet(pktio_entry_t *entry, const uint8_t *base, if (cos == NULL) return -EINVAL; - if (cos->s.queue == NULL || cos->s.pool == ODP_POOL_INVALID) + if (cos->s.queue == QUEUE_NULL || cos->s.pool == ODP_POOL_INVALID) return -EFAULT; *pool = cos->s.pool; pkt_hdr->p.input_flags.dst_queue = 1; - pkt_hdr->dst_queue = queue_fn->to_ext(cos->s.queue); + pkt_hdr->dst_queue = cos->s.queue; return 0; } diff --git a/platform/linux-generic/odp_packet_io.c b/platform/linux-generic/odp_packet_io.c index 4dd28549..6ff35046 100644 --- a/platform/linux-generic/odp_packet_io.c +++ b/platform/linux-generic/odp_packet_io.c @@ -566,11 +566,9 @@ static inline int pktin_recv_buf(odp_pktin_queue_t queue, buf_hdr = packet_to_buf_hdr(pkt); if (pkt_hdr->p.input_flags.dst_queue) { - queue_t dst_queue; int ret; - dst_queue = queue_fn->from_ext(pkt_hdr->dst_queue); - ret = queue_fn->enq(dst_queue, buf_hdr); + ret = queue_fn->enq(pkt_hdr->dst_queue, buf_hdr); if (ret < 0) odp_packet_free(pkt); continue; commit e3fce2fca6067b03f7fd18c9a438d2058521fe11 Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Fri Jun 30 17:10:56 2017 +0300 linux-gen: sched: remove unused sched interface functions Removed functions that are no longer used. Also removed unused parameter from save_context function. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_schedule_if.h b/platform/linux-generic/include/odp_schedule_if.h index 9b283678..4cd8c3e5 100644 --- a/platform/linux-generic/include/odp_schedule_if.h +++ b/platform/linux-generic/include/odp_schedule_if.h @@ -35,7 +35,7 @@ typedef int (*schedule_term_local_fn_t)(void); typedef void (*schedule_order_lock_fn_t)(void); typedef void (*schedule_order_unlock_fn_t)(void); typedef unsigned (*schedule_max_ordered_locks_fn_t)(void); -typedef void (*schedule_save_context_fn_t)(uint32_t queue_index, void *ptr); +typedef void (*schedule_save_context_fn_t)(uint32_t queue_index); typedef struct schedule_fn_t { int status_sync; @@ -68,11 +68,6 @@ extern const schedule_fn_t *sched_fn; int sched_cb_pktin_poll(int pktio_index, int num_queue, int index[]); void sched_cb_pktio_stop_finalize(int pktio_index); int sched_cb_num_pktio(void); -int sched_cb_num_queues(void); -int sched_cb_queue_prio(uint32_t queue_index); -int sched_cb_queue_grp(uint32_t queue_index); -int sched_cb_queue_is_ordered(uint32_t queue_index); -int sched_cb_queue_is_atomic(uint32_t queue_index); odp_queue_t sched_cb_queue_handle(uint32_t queue_index); void sched_cb_queue_destroy_finalize(uint32_t queue_index); int sched_cb_queue_deq_multi(uint32_t queue_index, odp_event_t ev[], int num); diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c index d907779b..4c85027b 100644 --- a/platform/linux-generic/odp_queue.c +++ b/platform/linux-generic/odp_queue.c @@ -520,7 +520,7 @@ static inline int deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], queue->s.tail = NULL; if (status_sync && queue->s.type == ODP_QUEUE_TYPE_SCHED) - sched_fn->save_context(queue->s.index, queue); + sched_fn->save_context(queue->s.index); UNLOCK(&queue->s.lock); @@ -672,25 +672,6 @@ static int queue_info(odp_queue_t handle, odp_queue_info_t *info) return 0; } -int sched_cb_num_queues(void) -{ - return ODP_CONFIG_QUEUES; -} - -int sched_cb_queue_prio(uint32_t queue_index) -{ - queue_entry_t *qe = get_qentry(queue_index); - - return qe->s.param.sched.prio; -} - -int sched_cb_queue_grp(uint32_t queue_index) -{ - queue_entry_t *qe = get_qentry(queue_index); - - return qe->s.param.sched.group; -} - odp_queue_t sched_cb_queue_handle(uint32_t queue_index) { return queue_from_id(queue_index); diff --git a/platform/linux-generic/odp_schedule_iquery.c b/platform/linux-generic/odp_schedule_iquery.c index c46e700b..d374dedd 100644 --- a/platform/linux-generic/odp_schedule_iquery.c +++ b/platform/linux-generic/odp_schedule_iquery.c @@ -1305,10 +1305,8 @@ static inline bool is_ordered_queue(unsigned int queue_index) return (sched->queues[queue_index].sync == ODP_SCHED_SYNC_ORDERED); } -static void schedule_save_context(uint32_t queue_index, void *ptr) +static void schedule_save_context(uint32_t queue_index) { - (void)ptr; - if (is_atomic_queue(queue_index)) { thread_local.atomic = &sched->availables[queue_index]; } else if (is_ordered_queue(queue_index)) { commit 29ad363b5ab3c5e6569f20ddf63e64cc53eeb019 Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Fri Jun 30 17:10:55 2017 +0300 linux-gen: sched: remove most dependecies to qentry Moved ordered queue context structure from queue internal structure to the scheduler. Ordering is a scheduler feature and thus all data and code about ordering should be in scheduler implementation. This removes most dependencies to qentry from the scheduler. Remaining dependecies are due to queue interface definition, which is not changed in this patch. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_queue_internal.h b/platform/linux-generic/include/odp_queue_internal.h index d79abd23..032dde88 100644 --- a/platform/linux-generic/include/odp_queue_internal.h +++ b/platform/linux-generic/include/odp_queue_internal.h @@ -42,13 +42,6 @@ struct queue_entry_s { odp_buffer_hdr_t *tail; int status; - struct { - odp_atomic_u64_t ctx; /**< Current ordered context id */ - odp_atomic_u64_t next_ctx; /**< Next unallocated context id */ - /** Array of ordered locks */ - odp_atomic_u64_t lock[CONFIG_QUEUE_MAX_ORD_LOCKS]; - } ordered ODP_ALIGNED_CACHE; - queue_enq_fn_t enqueue ODP_ALIGNED_CACHE; queue_deq_fn_t dequeue; queue_enq_multi_fn_t enqueue_multi; diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c index 2db95fc6..d907779b 100644 --- a/platform/linux-generic/odp_queue.c +++ b/platform/linux-generic/odp_queue.c @@ -57,16 +57,6 @@ static inline odp_queue_t queue_from_id(uint32_t queue_id) return _odp_cast_scalar(odp_queue_t, queue_id + 1); } -static inline int queue_is_atomic(queue_entry_t *qe) -{ - return qe->s.param.sched.sync == ODP_SCHED_SYNC_ATOMIC; -} - -static inline int queue_is_ordered(queue_entry_t *qe) -{ - return qe->s.param.sched.sync == ODP_SCHED_SYNC_ORDERED; -} - queue_entry_t *get_qentry(uint32_t queue_id) { return &queue_tbl->queue[queue_id]; @@ -278,13 +268,6 @@ static int queue_destroy(odp_queue_t handle) ODP_ERR("queue \"%s\" not empty\n", queue->s.name); return -1; } - if (queue_is_ordered(queue) && - odp_atomic_load_u64(&queue->s.ordered.ctx) != - odp_atomic_load_u64(&queue->s.ordered.next_ctx)) { - UNLOCK(&queue->s.lock); - ODP_ERR("queue \"%s\" reorder incomplete\n", queue->s.name); - return -1; - } switch (queue->s.status) { case QUEUE_STATUS_READY: @@ -610,20 +593,9 @@ static int queue_init(queue_entry_t *queue, const char *name, if (queue->s.param.sched.lock_count > sched_fn->max_ordered_locks()) return -1; - if (param->type == ODP_QUEUE_TYPE_SCHED) { + if (param->type == ODP_QUEUE_TYPE_SCHED) queue->s.param.deq_mode = ODP_QUEUE_OP_DISABLED; - if (param->sched.sync == ODP_SCHED_SYNC_ORDERED) { - unsigned i; - - odp_atomic_init_u64(&queue->s.ordered.ctx, 0); - odp_atomic_init_u64(&queue->s.ordered.next_ctx, 0); - - for (i = 0; i < queue->s.param.sched.lock_count; i++) - odp_atomic_init_u64(&queue->s.ordered.lock[i], - 0); - } - } queue->s.type = queue->s.param.type; queue->s.enqueue = queue_int_enq; @@ -719,16 +691,6 @@ int sched_cb_queue_grp(uint32_t queue_index) return qe->s.param.sched.group; } -int sched_cb_queue_is_ordered(uint32_t queue_index) -{ - return queue_is_ordered(get_qentry(queue_index)); -} - -int sched_cb_queue_is_atomic(uint32_t queue_index) -{ - return queue_is_atomic(get_qentry(queue_index)); -} - odp_queue_t sched_cb_queue_handle(uint32_t queue_index) { return queue_from_id(queue_index); diff --git a/platform/linux-generic/odp_schedule.c b/platform/linux-generic/odp_schedule.c index 00452279..bbcc85a5 100644 --- a/platform/linux-generic/odp_schedule.c +++ b/platform/linux-generic/odp_schedule.c @@ -65,8 +65,11 @@ ODP_STATIC_ASSERT((ODP_SCHED_PRIO_NORMAL > 0) && /* Maximum number of pktio poll commands */ #define NUM_PKTIO_CMD (MAX_PKTIN * NUM_PKTIO) +/* Not a valid index */ +#define NULL_INDEX ((uint32_t)-1) + /* Not a valid poll command */ -#define PKTIO_CMD_INVALID ((uint32_t)-1) +#define PKTIO_CMD_INVALID NULL_INDEX /* Pktio command is free */ #define PKTIO_CMD_FREE PKTIO_CMD_INVALID @@ -90,7 +93,7 @@ ODP_STATIC_ASSERT((ODP_SCHED_PRIO_NORMAL > 0) && #define PRIO_QUEUE_MASK (PRIO_QUEUE_RING_SIZE - 1) /* Priority queue empty, not a valid queue index. */ -#define PRIO_QUEUE_EMPTY ((uint32_t)-1) +#define PRIO_QUEUE_EMPTY NULL_INDEX /* For best performance, the number of queues should be a power of two. */ ODP_STATIC_ASSERT(CHECK_IS_POWER2(ODP_CONFIG_QUEUES), @@ -127,7 +130,7 @@ ODP_STATIC_ASSERT((8 * sizeof(pri_mask_t)) >= QUEUES_PER_PRIO, /* Storage for stashed enqueue operation arguments */ typedef struct { odp_buffer_hdr_t *buf_hdr[QUEUE_MULTI_MAX]; - queue_entry_t *queue; + uint32_t queue_index; int num; } ordered_stash_t; @@ -152,7 +155,8 @@ typedef struct { odp_queue_t queue; odp_event_t ev_stash[MAX_DEQ]; struct { - queue_entry_t *src_queue; /**< Source queue entry */ + /* Source queue index */ + uint32_t src_queue; uint64_t ctx; /**< Ordered context id */ int stash_num; /**< Number of stashed enqueue operations */ uint8_t in_order; /**< Order status */ @@ -197,6 +201,19 @@ typedef struct { uint32_t cmd_index; } pktio_cmd_t; +/* Order context of a queue */ +typedef struct { + /* Current ordered context id */ + odp_atomic_u64_t ctx ODP_ALIGNED_CACHE; + + /* Next unallocated context id */ + odp_atomic_u64_t next_ctx; + + /* Array of ordered locks */ + odp_atomic_u64_t lock[CONFIG_QUEUE_MAX_ORD_LOCKS]; + +} order_context_t ODP_ALIGNED_CACHE; + typedef struct { pri_mask_t pri_mask[NUM_PRIO]; odp_spinlock_t mask_lock; @@ -230,6 +247,8 @@ typedef struct { int grp; int prio; int queue_per_prio; + int sync; + unsigned order_lock_count; } queue[ODP_CONFIG_QUEUES]; struct { @@ -237,6 +256,8 @@ typedef struct { int num_cmd; } pktio[NUM_PKTIO]; + order_context_t order[ODP_CONFIG_QUEUES]; + } sched_global_t; /* Global scheduler context */ @@ -259,6 +280,7 @@ static void sched_local_init(void) sched_local.thr = odp_thread_id(); sched_local.queue = ODP_QUEUE_INVALID; sched_local.queue_index = PRIO_QUEUE_EMPTY; + sched_local.ordered.src_queue = NULL_INDEX; id = sched_local.thr & (QUEUES_PER_PRIO - 1); @@ -488,16 +510,35 @@ static void pri_clr_queue(uint32_t queue_index, int prio) static int schedule_init_queue(uint32_t queue_index, const odp_schedule_param_t *sched_param) { + int i; int prio = sched_param->prio; pri_set_queue(queue_index, prio); sched->queue[queue_index].grp = sched_param->group; sched->queue[queue_index].prio = prio; sched->queue[queue_index].queue_per_prio = queue_per_prio(queue_index); + sched->queue[queue_index].sync = sched_param->sync; + sched->queue[queue_index].order_lock_count = sched_param->lock_count; + + odp_atomic_init_u64(&sched->order[queue_index].ctx, 0); + odp_atomic_init_u64(&sched->order[queue_index].next_ctx, 0); + + for (i = 0; i < CONFIG_QUEUE_MAX_ORD_LOCKS; i++) + odp_atomic_init_u64(&sched->order[queue_index].lock[i], 0); return 0; } +static inline int queue_is_atomic(uint32_t queue_index) +{ + return sched->queue[queue_index].sync == ODP_SCHED_SYNC_ATOMIC; +} + +static inline int queue_is_ordered(uint32_t queue_index) +{ + return sched->queue[queue_index].sync == ODP_SCHED_SYNC_ORDERED; +} + static void schedule_destroy_queue(uint32_t queue_index) { int prio = sched->queue[queue_index].prio; @@ -506,6 +547,11 @@ static void schedule_destroy_queue(uint32_t queue_index) sched->queue[queue_index].grp = 0; sched->queue[queue_index].prio = 0; sched->queue[queue_index].queue_per_prio = 0; + + if (queue_is_ordered(queue_index) && + odp_atomic_load_u64(&sched->order[queue_index].ctx) != + odp_atomic_load_u64(&sched->order[queue_index].next_ctx)) + ODP_ERR("queue reorder incomplete\n"); } static int poll_cmd_queue_idx(int pktio_index, int pktin_idx) @@ -606,20 +652,20 @@ static void schedule_release_atomic(void) } } -static inline int ordered_own_turn(queue_entry_t *queue) +static inline int ordered_own_turn(uint32_t queue_index) { uint64_t ctx; - ctx = odp_atomic_load_acq_u64(&queue->s.ordered.ctx); + ctx = odp_atomic_load_acq_u64(&sched->order[queue_index].ctx); return ctx == sched_local.ordered.ctx; } -static inline void wait_for_order(queue_entry_t *queue) +static inline void wait_for_order(uint32_t queue_index) { /* Busy loop to synchronize ordered processing */ while (1) { - if (ordered_own_turn(queue)) + if (ordered_own_turn(queue_index)) break; odp_cpu_pause(); } @@ -635,52 +681,54 @@ static inline void ordered_stash_release(void) int i; for (i = 0; i < sched_local.ordered.stash_num; i++) { - queue_entry_t *queue; + queue_entry_t *queue_entry; + uint32_t queue_index; odp_buffer_hdr_t **buf_hdr; int num; - queue = sched_local.ordered.stash[i].queue; + queue_index = sched_local.ordered.stash[i].queue_index; + queue_entry = get_qentry(queue_index); buf_hdr = sched_local.ordered.stash[i].buf_hdr; num = sched_local.ordered.stash[i].num; - queue_fn->enq_multi(qentry_to_int(queue), buf_hdr, num); + queue_fn->enq_multi(qentry_to_int(queue_entry), buf_hdr, num); } sched_local.ordered.stash_num = 0; } static inline void release_ordered(void) { + uint32_t qi; unsigned i; - queue_entry_t *queue; - queue = sched_local.ordered.src_queue; + qi = sched_local.ordered.src_queue; - wait_for_order(queue); + wait_for_order(qi); /* Release all ordered locks */ - for (i = 0; i < queue->s.param.sched.lock_count; i++) { + for (i = 0; i < sched->queue[qi].order_lock_count; i++) { if (!sched_local.ordered.lock_called.u8[i]) - odp_atomic_store_rel_u64(&queue->s.ordered.lock[i], + odp_atomic_store_rel_u64(&sched->order[qi].lock[i], sched_local.ordered.ctx + 1); } sched_local.ordered.lock_called.all = 0; - sched_local.ordered.src_queue = NULL; + sched_local.ordered.src_queue = NULL_INDEX; sched_local.ordered.in_order = 0; ordered_stash_release(); /* Next thread can continue processing */ - odp_atomic_add_rel_u64(&queue->s.ordered.ctx, 1); + odp_atomic_add_rel_u64(&sched->order[qi].ctx, 1); } static void schedule_release_ordered(void) { - queue_entry_t *queue; + uint32_t queue_index; - queue = sched_local.ordered.src_queue; + queue_index = sched_local.ordered.src_queue; - if (odp_unlikely(!queue || sched_local.num)) + if (odp_unlikely((queue_index == NULL_INDEX) || sched_local.num)) return; release_ordered(); @@ -688,7 +736,7 @@ static void schedule_release_ordered(void) static inline void schedule_release_context(void) { - if (sched_local.ordered.src_queue != NULL) + if (sched_local.ordered.src_queue != NULL_INDEX) release_ordered(); else schedule_release_atomic(); @@ -715,9 +763,9 @@ static int schedule_ord_enq_multi(queue_t q_int, void *buf_hdr[], int i; uint32_t stash_num = sched_local.ordered.stash_num; queue_entry_t *dst_queue = qentry_from_int(q_int); - queue_entry_t *src_queue = sched_local.ordered.src_queue; + uint32_t src_queue = sched_local.ordered.src_queue; - if (!sched_local.ordered.src_queue || sched_local.ordered.in_order) + if ((src_queue == NULL_INDEX) || sched_local.ordered.in_order) return 0; if (ordered_own_turn(src_queue)) { @@ -740,7 +788,7 @@ static int schedule_ord_enq_multi(queue_t q_int, void *buf_hdr[], return 0; } - sched_local.ordered.stash[stash_num].queue = dst_queue; + sched_local.ordered.stash[stash_num].queue_index = dst_queue->s.index; sched_local.ordered.stash[stash_num].num = num; for (i = 0; i < num; i++) sched_local.ordered.stash[stash_num].buf_hdr[i] = buf_hdr[i]; @@ -803,7 +851,7 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], prio > ODP_SCHED_PRIO_DEFAULT)) max_deq = MAX_DEQ / 2; - ordered = sched_cb_queue_is_ordered(qi); + ordered = queue_is_ordered(qi); /* Do not cache ordered events locally to improve * parallelism. Ordered context can only be released @@ -835,21 +883,18 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], if (ordered) { uint64_t ctx; - queue_entry_t *queue; odp_atomic_u64_t *next_ctx; - queue = get_qentry(qi); - next_ctx = &queue->s.ordered.next_ctx; - + next_ctx = &sched->order[qi].next_ctx; ctx = odp_atomic_fetch_inc_u64(next_ctx); sched_local.ordered.ctx = ctx; - sched_local.ordered.src_queue = queue; + sched_local.ordered.src_queue = qi; /* Continue scheduling ordered queues */ ring_enq(ring, PRIO_QUEUE_MASK, qi); - } else if (sched_cb_queue_is_atomic(qi)) { + } else if (queue_is_atomic(qi)) { /* Hold queue during atomic access */ sched_local.queue_index = qi; } else { @@ -1039,14 +1084,14 @@ static int schedule_multi(odp_queue_t *out_queue, uint64_t wait, static inline void order_lock(void) { - queue_entry_t *queue; + uint32_t queue_index; - queue = sched_local.ordered.src_queue; + queue_index = sched_local.ordered.src_queue; - if (!queue) + if (queue_index == NULL_INDEX) return; - wait_for_order(queue); + wait_for_order(queue_index); } static void order_unlock(void) @@ -1056,14 +1101,15 @@ static void order_unlock(void) static void schedule_order_lock(unsigned lock_index) { odp_atomic_u64_t *ord_lock; - queue_entry_t *queue; + uint32_t queue_index; - queue = sched_local.ordered.src_queue; + queue_index = sched_local.ordered.src_queue; - ODP_ASSERT(queue && lock_index <= queue->s.param.sched.lock_count && + ODP_ASSERT(queue_index != NULL_INDEX && + lock_index <= sched->queue[queue_index].order_lock_count && !sched_local.ordered.lock_called.u8[lock_index]); - ord_lock = &queue->s.ordered.lock[lock_index]; + ord_lock = &sched->order[queue_index].lock[lock_index]; /* Busy loop to synchronize ordered processing */ while (1) { @@ -1082,13 +1128,14 @@ static void schedule_order_lock(unsigned lock_index) static void schedule_order_unlock(unsigned lock_index) { odp_atomic_u64_t *ord_lock; - queue_entry_t *queue; + uint32_t queue_index; - queue = sched_local.ordered.src_queue; + queue_index = sched_local.ordered.src_queue; - ODP_ASSERT(queue && lock_index <= queue->s.param.sched.lock_count); + ODP_ASSERT(queue_index != NULL_INDEX && + lock_index <= sched->queue[queue_index].order_lock_count); - ord_lock = &queue->s.ordered.lock[lock_index]; + ord_lock = &sched->order[queue_index].lock[lock_index]; ODP_ASSERT(sched_local.ordered.ctx == odp_atomic_load_u64(ord_lock)); diff --git a/platform/linux-generic/odp_schedule_iquery.c b/platform/linux-generic/odp_schedule_iquery.c index fd0ca5e8..c46e700b 100644 --- a/platform/linux-generic/odp_schedule_iquery.c +++ b/platform/linux-generic/odp_schedule_iquery.c @@ -70,6 +70,8 @@ typedef struct { /* Maximum number of pktio poll commands */ #define NUM_PKTIO_CMD (MAX_PKTIN * NUM_PKTIO) +/* Not a valid index */ +#define NULL_INDEX ((uint32_t)-1) /* Pktio command is free */ #define PKTIO_CMD_FREE ((uint32_t)-1) @@ -116,6 +118,19 @@ typedef struct { /* Forward declaration */ typedef struct sched_thread_local sched_thread_local_t; +/* Order context of a queue */ +typedef struct { + /* Current ordered context id */ + odp_atomic_u64_t ctx ODP_ALIGNED_CACHE; + + /* Next unallocated context id */ + odp_atomic_u64_t next_ctx; + + /* Array of ordered locks */ + odp_atomic_u64_t lock[CONFIG_QUEUE_MAX_ORD_LOCKS]; + +} order_context_t ODP_ALIGNED_CACHE; + typedef struct { odp_shm_t selfie; @@ -138,6 +153,8 @@ typedef struct { /* Quick reference to per thread context */ sched_thread_local_t *threads[ODP_THREAD_COUNT_MAX]; + + order_context_t order[ODP_CONFIG_QUEUES]; } sched_global_t; /* Per thread events cache */ @@ -153,7 +170,7 @@ typedef struct { /* Storage for stashed enqueue operation arguments */ typedef struct { odp_buffer_hdr_t *buf_hdr[QUEUE_MULTI_MAX]; - queue_entry_t *queue; + uint32_t queue_index; int num; } ordered_stash_t; @@ -194,7 +211,8 @@ struct sched_thread_local { sparse_bitmap_iterator_t iterators[NUM_SCHED_PRIO]; struct { - queue_entry_t *src_queue; /**< Source queue entry */ + /* Source queue index */ + uint32_t src_queue; uint64_t ctx; /**< Ordered context id */ int stash_num; /**< Number of stashed enqueue operations */ uint8_t in_order; /**< Order status */ @@ -313,6 +331,7 @@ static void sched_thread_local_reset(void) thread_local.thread = odp_thread_id(); thread_local.cache.queue = ODP_QUEUE_INVALID; + thread_local.ordered.src_queue = NULL_INDEX; odp_rwlock_init(&thread_local.lock); @@ -394,7 +413,7 @@ static int schedule_term_local(void) static int init_sched_queue(uint32_t queue_index, const odp_schedule_param_t *sched_param) { - int prio, group, thread; + int prio, group, thread, i; sched_prio_t *P; sched_group_t *G; sched_thread_local_t *local; @@ -427,6 +446,12 @@ static int init_sched_queue(uint32_t queue_index, memcpy(&sched->queues[queue_index], sched_param, sizeof(odp_schedule_param_t)); + odp_atomic_init_u64(&sched->order[queue_index].ctx, 0); + odp_atomic_init_u64(&sched->order[queue_index].next_ctx, 0); + + for (i = 0; i < CONFIG_QUEUE_MAX_ORD_LOCKS; i++) + odp_atomic_init_u64(&sched->order[queue_index].lock[i], 0); + /* Update all threads in this schedule group to * start check this queue index upon scheduling. */ @@ -501,6 +526,11 @@ static void destroy_sched_queue(uint32_t queue_index) __destroy_sched_queue(G, queue_index); odp_rwlock_write_unlock(&G->lock); + + if (sched->queues[queue_index].sync == ODP_SCHED_SYNC_ORDERED && + odp_atomic_load_u64(&sched->order[queue_index].ctx) != + odp_atomic_load_u64(&sched->order[queue_index].next_ctx)) + ODP_ERR("queue reorder incomplete\n"); } static int pktio_cmd_queue_hash(int pktio, int pktin) @@ -1067,20 +1097,20 @@ static void schedule_release_atomic(void) } } -static inline int ordered_own_turn(queue_entry_t *queue) +static inline int ordered_own_turn(uint32_t queue_index) { uint64_t ctx; - ctx = odp_atomic_load_acq_u64(&queue->s.ordered.ctx); + ctx = odp_atomic_load_acq_u64(&sched->order[queue_index].ctx); return ctx == thread_local.ordered.ctx; } -static inline void wait_for_order(queue_entry_t *queue) +static inline void wait_for_order(uint32_t queue_index) { /* Busy loop to synchronize ordered processing */ while (1) { - if (ordered_own_turn(queue)) + if (ordered_own_turn(queue_index)) break; odp_cpu_pause(); } @@ -1096,52 +1126,55 @@ static inline void ordered_stash_release(void) int i; for (i = 0; i < thread_local.ordered.stash_num; i++) { - queue_entry_t *queue; + queue_entry_t *queue_entry; + uint32_t queue_index; odp_buffer_hdr_t **buf_hdr; int num; - queue = thread_local.ordered.stash[i].queue; + queue_index = thread_local.ordered.stash[i].queue_index; + queue_entry = get_qentry(queue_index); buf_hdr = thread_local.ordered.stash[i].buf_hdr; num = thread_local.ordered.stash[i].num; - queue_fn->enq_multi(qentry_to_int(queue), buf_hdr, num); + queue_fn->enq_multi(qentry_to_int(queue_entry), buf_hdr, num); } thread_local.ordered.stash_num = 0; } static inline void release_ordered(void) { + uint32_t qi; unsigned i; - queue_entry_t *queue; - queue = thread_local.ordered.src_queue; + qi = thread_local.ordered.src_queue; - wait_for_order(queue); + wait_for_order(qi); /* Release all ordered locks */ - for (i = 0; i < queue->s.param.sched.lock_count; i++) { + for (i = 0; i < sched->queues[qi].lock_count; i++) { if (!thread_local.ordered.lock_called.u8[i]) - odp_atomic_store_rel_u64(&queue->s.ordered.lock[i], + odp_atomic_store_rel_u64(&sched->order[qi].lock[i], thread_local.ordered.ctx + 1); } thread_local.ordered.lock_called.all = 0; - thread_local.ordered.src_queue = NULL; + thread_local.ordered.src_queue = NULL_INDEX; thread_local.ordered.in_order = 0; ordered_stash_release(); /* Next thread can continue processing */ - odp_atomic_add_rel_u64(&queue->s.ordered.ctx, 1); + odp_atomic_add_rel_u64(&sched->order[qi].ctx, 1); } static void schedule_release_ordered(void) { - queue_entry_t *queue; + uint32_t queue_index; - queue = thread_local.ordered.src_queue; + queue_index = thread_local.ordered.src_queue; - if (odp_unlikely(!queue || thread_local.cache.count)) + if (odp_unlikely((queue_index == NULL_INDEX) || + thread_local.cache.count)) return; release_ordered(); @@ -1149,7 +1182,7 @@ static void schedule_release_ordered(void) static inline void schedule_release_context(void) { - if (thread_local.ordered.src_queue != NULL) + if (thread_local.ordered.src_queue != NULL_INDEX) release_ordered(); else schedule_release_atomic(); @@ -1161,9 +1194,9 @@ static int schedule_ord_enq_multi(queue_t q_int, void *buf_hdr[], int i; uint32_t stash_num = thread_local.ordered.stash_num; queue_entry_t *dst_queue = qentry_from_int(q_int); - queue_entry_t *src_queue = thread_local.ordered.src_queue; + uint32_t src_queue = thread_local.ordered.src_queue; - if (!thread_local.ordered.src_queue || thread_local.ordered.in_order) + if ((src_queue == NULL_INDEX) || thread_local.ordered.in_order) return 0; if (ordered_own_turn(src_queue)) { @@ -1186,7 +1219,7 @@ static int schedule_ord_enq_multi(queue_t q_int, void *buf_hdr[], return 0; } - thread_local.ordered.stash[stash_num].queue = dst_queue; + thread_local.ordered.stash[stash_num].queue_index = dst_queue->s.index; thread_local.ordered.stash[stash_num].num = num; for (i = 0; i < num; i++) thread_local.ordered.stash[stash_num].buf_hdr[i] = buf_hdr[i]; @@ -1199,14 +1232,14 @@ static int schedule_ord_enq_multi(queue_t q_int, void *buf_hdr[], static void order_lock(void) { - queue_entry_t *queue; + uint32_t queue_index; - queue = thread_local.ordered.src_queue; + queue_index = thread_local.ordered.src_queue; - if (!queue) + if (queue_index == NULL_INDEX) return; - wait_for_order(queue); + wait_for_order(queue_index); } static void order_unlock(void) @@ -1216,14 +1249,15 @@ static void order_unlock(void) static void schedule_order_lock(unsigned lock_index) { odp_atomic_u64_t *ord_lock; - queue_entry_t *queue; + uint32_t queue_index; - queue = thread_local.ordered.src_queue; + queue_index = thread_local.ordered.src_queue; - ODP_ASSERT(queue && lock_index <= queue->s.param.sched.lock_count && + ODP_ASSERT(queue_index != NULL_INDEX && + lock_index <= sched->queues[queue_index].lock_count && !thread_local.ordered.lock_called.u8[lock_index]); - ord_lock = &queue->s.ordered.lock[lock_index]; + ord_lock = &sched->order[queue_index].lock[lock_index]; /* Busy loop to synchronize ordered processing */ while (1) { @@ -1242,13 +1276,14 @@ static void schedule_order_lock(unsigned lock_index) static void schedule_order_unlock(unsigned lock_index) { odp_atomic_u64_t *ord_lock; - queue_entry_t *queue; + uint32_t queue_index; - queue = thread_local.ordered.src_queue; + queue_index = thread_local.ordered.src_queue; - ODP_ASSERT(queue && lock_index <= queue->s.param.sched.lock_count); + ODP_ASSERT(queue_index != NULL_INDEX && + lock_index <= sched->queues[queue_index].lock_count); - ord_lock = &queue->s.ordered.lock[lock_index]; + ord_lock = &sched->order[queue_index].lock[lock_index]; ODP_ASSERT(thread_local.ordered.ctx == odp_atomic_load_u64(ord_lock)); @@ -1272,7 +1307,7 @@ static inline bool is_ordered_queue(unsigned int queue_index) static void schedule_save_context(uint32_t queue_index, void *ptr) { - queue_entry_t *queue = ptr; + (void)ptr; if (is_atomic_queue(queue_index)) { thread_local.atomic = &sched->availables[queue_index]; @@ -1280,11 +1315,11 @@ static void schedule_save_context(uint32_t queue_index, void *ptr) uint64_t ctx; odp_atomic_u64_t *next_ctx; - next_ctx = &queue->s.ordered.next_ctx; + next_ctx = &sched->order[queue_index].next_ctx; ctx = odp_atomic_fetch_inc_u64(next_ctx); thread_local.ordered.ctx = ctx; - thread_local.ordered.src_queue = queue; + thread_local.ordered.src_queue = queue_index; } } commit 8ef745b7b78b49fc58d14e7b79b450c0bb560558 Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Fri Jun 30 17:10:54 2017 +0300 linux-gen: sched: use config max ordered locks Use config file value for the number of ordered locks everywhere. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_config_internal.h b/platform/linux-generic/include/odp_config_internal.h index e7d84c90..292be356 100644 --- a/platform/linux-generic/include/odp_config_internal.h +++ b/platform/linux-generic/include/odp_config_internal.h @@ -24,7 +24,7 @@ extern "C" { /* * Maximum number of ordered locks per queue */ -#define CONFIG_QUEUE_MAX_ORD_LOCKS 4 +#define CONFIG_QUEUE_MAX_ORD_LOCKS 2 /* * Maximum number of packet IO resources diff --git a/platform/linux-generic/odp_schedule.c b/platform/linux-generic/odp_schedule.c index 3a462995..00452279 100644 --- a/platform/linux-generic/odp_schedule.c +++ b/platform/linux-generic/odp_schedule.c @@ -121,12 +121,6 @@ ODP_STATIC_ASSERT((8 * sizeof(pri_mask_t)) >= QUEUES_PER_PRIO, /* Maximum number of dequeues */ #define MAX_DEQ CONFIG_BURST_SIZE -/* Maximum number of ordered locks per queue */ -#define MAX_ORDERED_LOCKS_PER_QUEUE 2 - -ODP_STATIC_ASSERT(MAX_ORDERED_LOCKS_PER_QUEUE <= CONFIG_QUEUE_MAX_ORD_LOCKS, - "Too_many_ordered_locks"); - /* Ordered stash size */ #define MAX_ORDERED_STASH 512 @@ -449,7 +443,7 @@ static inline int grp_update_tbl(void) static unsigned schedule_max_ordered_locks(void) { - return MAX_ORDERED_LOCKS_PER_QUEUE; + return CONFIG_QUEUE_MAX_ORD_LOCKS; } static inline int queue_per_prio(uint32_t queue_index) diff --git a/platform/linux-generic/odp_schedule_iquery.c b/platform/linux-generic/odp_schedule_iquery.c index 6eea24c6..fd0ca5e8 100644 --- a/platform/linux-generic/odp_schedule_iquery.c +++ b/platform/linux-generic/odp_schedule_iquery.c @@ -147,12 +147,6 @@ typedef struct { odp_event_t stash[MAX_DEQ], *top; } event_cache_t; -/* Maximum number of ordered locks per queue */ -#define MAX_ORDERED_LOCKS_PER_QUEUE 2 - -ODP_STATIC_ASSERT(MAX_ORDERED_LOCKS_PER_QUEUE <= CONFIG_QUEUE_MAX_ORD_LOCKS, - "Too_many_ordered_locks"); - /* Ordered stash size */ #define MAX_ORDERED_STASH 512 @@ -1263,7 +1257,7 @@ static void schedule_order_unlock(unsigned lock_index) static unsigned schedule_max_ordered_locks(void) { - return MAX_ORDERED_LOCKS_PER_QUEUE; + return CONFIG_QUEUE_MAX_ORD_LOCKS; } static inline bool is_atomic_queue(unsigned int queue_index) commit d9c85310cb82f83010c839b6a1bbb5a1fa672ceb Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Fri Jun 30 17:10:53 2017 +0300 linux-gen: sched: remove schedule interface depedency to qentry Do not use queue internal type in schedule interface. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_schedule_if.h b/platform/linux-generic/include/odp_schedule_if.h index 5d10cd37..9b283678 100644 --- a/platform/linux-generic/include/odp_schedule_if.h +++ b/platform/linux-generic/include/odp_schedule_if.h @@ -35,9 +35,10 @@ typedef int (*schedule_term_local_fn_t)(void); typedef void (*schedule_order_lock_fn_t)(void); typedef void (*schedule_order_unlock_fn_t)(void); typedef unsigned (*schedule_max_ordered_locks_fn_t)(void); -typedef void (*schedule_save_context_fn_t)(queue_entry_t *queue); +typedef void (*schedule_save_context_fn_t)(uint32_t queue_index, void *ptr); typedef struct schedule_fn_t { + int status_sync; schedule_pktio_start_fn_t pktio_start; schedule_thr_add_fn_t thr_add; schedule_thr_rem_fn_t thr_rem; @@ -45,7 +46,6 @@ typedef struct schedule_fn_t { schedule_init_queue_fn_t init_queue; schedule_destroy_queue_fn_t destroy_queue; schedule_sched_queue_fn_t sched_queue; - schedule_unsched_queue_fn_t unsched_queue; schedule_ord_enq_multi_fn_t ord_enq_multi; schedule_init_global_fn_t init_global; schedule_term_global_fn_t term_global; @@ -54,7 +54,11 @@ typedef struct schedule_fn_t { schedule_order_lock_fn_t order_lock; schedule_order_unlock_fn_t order_unlock; schedule_max_ordered_locks_fn_t max_ordered_locks; + + /* Called only when status_sync is set */ + schedule_unsched_queue_fn_t unsched_queue; schedule_save_context_fn_t save_context; + } schedule_fn_t; /* Interface towards the scheduler */ diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c index 19945584..2db95fc6 100644 --- a/platform/linux-generic/odp_queue.c +++ b/platform/linux-generic/odp_queue.c @@ -474,6 +474,7 @@ static inline int deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int i, j; queue_entry_t *queue; int updated = 0; + int status_sync = sched_fn->status_sync; queue = qentry_from_int(q_int); LOCK(&queue->s.lock); @@ -490,7 +491,9 @@ static inline int deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], /* Already empty queue */ if (queue->s.status == QUEUE_STATUS_SCHED) { queue->s.status = QUEUE_STATUS_NOTSCHED; - sched_fn->unsched_queue(queue->s.index); + + if (status_sync) + sched_fn->unsched_queue(queue->s.index); } UNLOCK(&queue->s.lock); @@ -533,8 +536,8 @@ static inline int deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], if (hdr == NULL) queue->s.tail = NULL; - if (queue->s.type == ODP_QUEUE_TYPE_SCHED) - sched_fn->save_context(queue); + if (status_sync && queue->s.type == ODP_QUEUE_TYPE_SCHED) + sched_fn->save_context(queue->s.index, queue); UNLOCK(&queue->s.lock); diff --git a/platform/linux-generic/odp_schedule.c b/platform/linux-generic/odp_schedule.c index 92634235..3a462995 100644 --- a/platform/linux-generic/odp_schedule.c +++ b/platform/linux-generic/odp_schedule.c @@ -22,6 +22,9 @@ #include <odp/api/sync.h> #include <odp/api/packet_io.h> #include <odp_ring_internal.h> +#include <odp_timer_internal.h> + +/* Should remove this dependency */ #include <odp_queue_internal.h> /* Number of priority levels */ @@ -1337,22 +1340,14 @@ static int schedule_sched_queue(uint32_t queue_index) return 0; } -static int schedule_unsched_queue(uint32_t queue_index ODP_UNUSED) -{ - return 0; -} - static int schedule_num_grps(void) { return NUM_SCHED_GRPS; } -static void schedule_save_context(queue_entry_t *queue ODP_UNUSED) -{ -} - /* Fill in scheduler interface */ const schedule_fn_t schedule_default_fn = { + .status_sync = 0, .pktio_start = schedule_pktio_start, .thr_add = schedule_thr_add, .thr_rem = schedule_thr_rem, @@ -1360,7 +1355,6 @@ const schedule_fn_t schedule_default_fn = { .init_queue = schedule_init_queue, .destroy_queue = schedule_destroy_queue, .sched_queue = schedule_sched_queue, - .unsched_queue = schedule_unsched_queue, .ord_enq_multi = schedule_ord_enq_multi, .init_global = schedule_init_global, .term_global = schedule_term_global, @@ -1369,7 +1363,8 @@ const schedule_fn_t schedule_default_fn = { .order_lock = order_lock, .order_unlock = order_unlock, .max_ordered_locks = schedule_max_ordered_locks, - .save_context = schedule_save_context + .unsched_queue = NULL, + .save_context = NULL }; /* Fill in scheduler API calls */ diff --git a/platform/linux-generic/odp_schedule_iquery.c b/platform/linux-generic/odp_schedule_iquery.c index 79086843..6eea24c6 100644 --- a/platform/linux-generic/odp_schedule_iquery.c +++ b/platform/linux-generic/odp_schedule_iquery.c @@ -12,7 +12,6 @@ #include <odp_internal.h> #include <odp_debug_internal.h> #include <odp_ring_internal.h> -#include <odp_queue_internal.h> #include <odp_buffer_internal.h> #include <odp_bitmap_internal.h> #include <odp/api/thread.h> @@ -24,6 +23,9 @@ #include <odp/api/packet_io.h> #include <odp_config_internal.h> +/* Should remove this dependency */ +#include <odp_queue_internal.h> + /* Number of priority levels */ #define NUM_SCHED_PRIO 8 @@ -1264,11 +1266,23 @@ static unsigned schedule_max_ordered_locks(void) return MAX_ORDERED_LOCKS_PER_QUEUE; } -static void schedule_save_context(queue_entry_t *queue) +static inline bool is_atomic_queue(unsigned int queue_index) +{ + return (sched->queues[queue_index].sync == ODP_SCHED_SYNC_ATOMIC); +} + +static inline bool is_ordered_queue(unsigned int queue_index) +{ + return (sched->queues[queue_index].sync == ODP_SCHED_SYNC_ORDERED); +} + +static void schedule_save_context(uint32_t queue_index, void *ptr) { - if (queue->s.param.sched.sync == ODP_SCHED_SYNC_ATOMIC) { - thread_local.atomic = &sched->availables[queue->s.index]; - } else if (queue->s.param.sched.sync == ODP_SCHED_SYNC_ORDERED) { + queue_entry_t *queue = ptr; + + if (is_atomic_queue(queue_index)) { + thread_local.atomic = &sched->availables[queue_index]; + } else if (is_ordered_queue(queue_index)) { uint64_t ctx; odp_atomic_u64_t *next_ctx; @@ -1282,6 +1296,7 @@ static void schedule_save_context(queue_entry_t *queue) /* Fill in scheduler interface */ const schedule_fn_t schedule_iquery_fn = { + .status_sync = 1, .pktio_start = schedule_pktio_start, .thr_add = group_add_thread, .thr_rem = group_remove_thread, @@ -1289,7 +1304,6 @@ const schedule_fn_t schedule_iquery_fn = { .init_queue = init_sched_queue, .destroy_queue = destroy_sched_queue, .sched_queue = schedule_sched_queue, - .unsched_queue = schedule_unsched_queue, .ord_enq_multi = schedule_ord_enq_multi, .init_global = schedule_init_global, .term_global = schedule_term_global, @@ -1298,7 +1312,8 @@ const schedule_fn_t schedule_iquery_fn = { .order_lock = order_lock, .order_unlock = order_unlock, .max_ordered_locks = schedule_max_ordered_locks, - .save_context = schedule_save_context, + .unsched_queue = schedule_unsched_queue, + .save_context = schedule_save_context }; /* Fill in scheduler API calls */ @@ -1425,18 +1440,6 @@ static void thread_clear_interests(sched_thread_local_t *thread, } } -static inline bool is_atomic_queue(unsigned int queue_index) -{ - return (sched->queues[queue_index].sync - == ODP_SCHED_SYNC_ATOMIC); -} - -static inline bool is_ordered_queue(unsigned int queue_index) -{ - return (sched->queues[queue_index].sync - == ODP_SCHED_SYNC_ORDERED); -} - static inline bool compete_atomic_queue(unsigned int queue_index) { bool expected = sched->availables[queue_index]; diff --git a/platform/linux-generic/odp_schedule_sp.c b/platform/linux-generic/odp_schedule_sp.c index 66715c13..d2ba539c 100644 --- a/platform/linux-generic/odp_schedule_sp.c +++ b/platform/linux-generic/odp_schedule_sp.c @@ -409,11 +409,6 @@ static int sched_queue(uint32_t qi) return 0; } -static int unsched_queue(uint32_t qi ODP_UNUSED) -{ - return 0; -} - static int ord_enq_multi(queue_t q_int, void *buf_hdr[], int num, int *ret) { @@ -827,12 +822,9 @@ static void order_unlock(void) { } -static void save_context(queue_entry_t *queue ODP_UNUSED) -{ -} - /* Fill in scheduler interface */ const schedule_fn_t schedule_sp_fn = { + .status_sync = 0, .pktio_start = pktio_start, .thr_add = thr_add, .thr_rem = thr_rem, @@ -840,16 +832,16 @@ const schedule_fn_t schedule_sp_fn = { .init_queue = init_queue, .destroy_queue = destroy_queue, .sched_queue = sched_queue, - .unsched_queue = unsched_queue, .ord_enq_multi = ord_enq_multi, .init_global = init_global, .term_global = term_global, .init_local = init_local, .term_local = term_local, - .order_lock = order_lock, - .order_unlock = order_unlock, + .order_lock = order_lock, + .order_unlock = order_unlock, .max_ordered_locks = max_ordered_locks, - .save_context = save_context + .unsched_queue = NULL, + .save_context = NULL }; /* Fill in scheduler API calls */ commit 5a38050f431d904b7354cd50611304da0a601727 Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Mon Jun 12 14:11:33 2017 +0300 linux-gen: queue: clean up after modular interface Clean up function and parameter naming after modular interface patch. Queue_t type is referred as "queue internal": queue_int or q_int. Term "handle" is reserved for API level handles (e.g. odp_queue_t, odp_pktio_t, etc) through out linux-gen implementation. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_queue_if.h b/platform/linux-generic/include/odp_queue_if.h index 4359435f..168d0e9e 100644 --- a/platform/linux-generic/include/odp_queue_if.h +++ b/platform/linux-generic/include/odp_queue_if.h @@ -53,24 +53,24 @@ typedef int (*queue_term_global_fn_t)(void); typedef int (*queue_init_local_fn_t)(void); typedef int (*queue_term_local_fn_t)(void); typedef queue_t (*queue_from_ext_fn_t)(odp_queue_t handle); -typedef odp_queue_t (*queue_to_ext_fn_t)(queue_t handle); -typedef int (*queue_enq_fn_t)(queue_t handle, odp_buffer_hdr_t *); -typedef int (*queue_enq_multi_fn_t)(queue_t handle, odp_buffer_hdr_t **, int); -typedef odp_buffer_hdr_t *(*queue_deq_fn_t)(queue_t handle); -typedef int (*queue_deq_multi_fn_t)(queue_t handle, odp_buffer_hdr_t **, int); -typedef odp_pktout_queue_t (*queue_get_pktout_fn_t)(queue_t handle); -typedef void (*queue_set_pktout_fn_t)(queue_t handle, odp_pktio_t pktio, +typedef odp_queue_t (*queue_to_ext_fn_t)(queue_t q_int); +typedef int (*queue_enq_fn_t)(queue_t q_int, odp_buffer_hdr_t *); +typedef int (*queue_enq_multi_fn_t)(queue_t q_int, odp_buffer_hdr_t **, int); +typedef odp_buffer_hdr_t *(*queue_deq_fn_t)(queue_t q_int); +typedef int (*queue_deq_multi_fn_t)(queue_t q_int, odp_buffer_hdr_t **, int); +typedef odp_pktout_queue_t (*queue_get_pktout_fn_t)(queue_t q_int); +typedef void (*queue_set_pktout_fn_t)(queue_t q_int, odp_pktio_t pktio, int index); -typedef odp_pktin_queue_t (*queue_get_pktin_fn_t)(queue_t handle); -typedef void (*queue_set_pktin_fn_t)(queue_t handle, odp_pktio_t pktio, +typedef odp_pktin_queue_t (*queue_get_pktin_fn_t)(queue_t q_int); +typedef void (*queue_set_pktin_fn_t)(queue_t q_int, odp_pktio_t pktio, int index); -typedef void (*queue_set_enq_fn_t)(queue_t handle, queue_enq_fn_t func); -typedef void (*queue_set_enq_multi_fn_t)(queue_t handle, +typedef void (*queue_set_enq_fn_t)(queue_t q_int, queue_enq_fn_t func); +typedef void (*queue_set_enq_multi_fn_t)(queue_t q_int, queue_enq_multi_fn_t func); -typedef void (*queue_set_deq_fn_t)(queue_t handle, queue_deq_fn_t func); -typedef void (*queue_set_deq_multi_fn_t)(queue_t handle, +typedef void (*queue_set_deq_fn_t)(queue_t q_int, queue_deq_fn_t func); +typedef void (*queue_set_deq_multi_fn_t)(queue_t q_int, queue_deq_multi_fn_t func); -typedef void (*queue_set_type_fn_t)(queue_t handle, odp_queue_type_t type); +typedef void (*queue_set_type_fn_t)(queue_t q_int, odp_queue_type_t type); /* Queue functions towards other internal components */ typedef struct { diff --git a/platform/linux-generic/include/odp_queue_internal.h b/platform/linux-generic/include/odp_queue_internal.h index 0c31ce8a..d79abd23 100644 --- a/platform/linux-generic/include/odp_queue_internal.h +++ b/platform/linux-generic/include/odp_queue_internal.h @@ -78,9 +78,9 @@ static inline uint32_t queue_to_id(odp_queue_t handle) return _odp_typeval(handle) - 1; } -static inline queue_entry_t *qentry_from_int(queue_t handle) +static inline queue_entry_t *qentry_from_int(queue_t q_int) { - return (queue_entry_t *)(void *)(handle); + return (queue_entry_t *)(void *)(q_int); } static inline queue_t qentry_to_int(queue_entry_t *qentry) diff --git a/platform/linux-generic/include/odp_schedule_if.h b/platform/linux-generic/include/odp_schedule_if.h index 9adacef7..5d10cd37 100644 --- a/platform/linux-generic/include/odp_schedule_if.h +++ b/platform/linux-generic/include/odp_schedule_if.h @@ -26,7 +26,7 @@ typedef int (*schedule_init_queue_fn_t)(uint32_t queue_index, typedef void (*schedule_destroy_queue_fn_t)(uint32_t queue_index); typedef int (*schedule_sched_queue_fn_t)(uint32_t queue_index); typedef int (*schedule_unsched_queue_fn_t)(uint32_t queue_index); -typedef int (*schedule_ord_enq_multi_fn_t)(queue_t handle, +typedef int (*schedule_ord_enq_multi_fn_t)(queue_t q_int, void *buf_hdr[], int num, int *ret); typedef int (*schedule_init_global_fn_t)(void); typedef int (*schedule_term_global_fn_t)(void); diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c index 3e18f578..19945584 100644 --- a/platform/linux-generic/odp_queue.c +++ b/platform/linux-generic/odp_queue.c @@ -35,20 +35,22 @@ #include <string.h> #include <inttypes.h> +static int queue_init(queue_entry_t *queue, const char *name, + const odp_queue_param_t *param); + typedef struct queue_table_t { queue_entry_t queue[ODP_CONFIG_QUEUES]; } queue_table_t; static queue_table_t *queue_tbl; -static queue_t queue_from_ext(odp_queue_t handle); -static int _queue_enq(queue_t handle, odp_buffer_hdr_t *buf_hdr); -static odp_buffer_hdr_t *_queue_deq(queue_t handle); +static inline queue_entry_t *handle_to_qentry(odp_queue_t handle) +{ + uint32_t queue_id; -static int _queue_enq_multi(queue_t handle, odp_buffer_hdr_t *buf_hdr[], - int num); -static int _queue_deq_multi(queue_t handle, odp_buffer_hdr_t *buf_hdr[], - int num); + queue_id = queue_to_id(handle); + return get_qentry(queue_id); +} static inline odp_queue_t queue_from_id(uint32_t queue_id) { @@ -70,50 +72,6 @@ queue_entry_t *get_qentry(uint32_t queue_id) return &queue_tbl->queue[queue_id]; } -static int queue_init(queue_entry_t *queue, const char *name, - const odp_queue_param_t *param) -{ - if (name == NULL) { - queue->s.name[0] = 0; - } else { - strncpy(queue->s.name, name, ODP_QUEUE_NAME_LEN - 1); - queue->s.name[ODP_QUEUE_NAME_LEN - 1] = 0; - } - memcpy(&queue->s.param, param, sizeof(odp_queue_param_t)); - if (queue->s.param.sched.lock_count > sched_fn->max_ordered_locks()) - return -1; - - if (param->type == ODP_QUEUE_TYPE_SCHED) { - queue->s.param.deq_mode = ODP_QUEUE_OP_DISABLED; - - if (param->sched.sync == ODP_SCHED_SYNC_ORDERED) { - unsigned i; - - odp_atomic_init_u64(&queue->s.ordered.ctx, 0); - odp_atomic_init_u64(&queue->s.ordered.next_ctx, 0); - - for (i = 0; i < queue->s.param.sched.lock_count; i++) - odp_atomic_init_u64(&queue->s.ordered.lock[i], - 0); - } - } - queue->s.type = queue->s.param.type; - - queue->s.enqueue = _queue_enq; - queue->s.dequeue = _queue_deq; - queue->s.enqueue_multi = _queue_enq_multi; - queue->s.dequeue_multi = _queue_deq_multi; - - queue->s.pktin = PKTIN_INVALID; - queue->s.pktout = PKTOUT_INVALID; - - queue->s.head = NULL; - queue->s.tail = NULL; - - return 0; -} - - static int queue_init_global(void) { uint32_t i; @@ -204,27 +162,27 @@ static int queue_capability(odp_queue_capability_t *capa) static odp_queue_type_t queue_type(odp_queue_t handle) { - return qentry_from_int(queue_from_ext(handle))->s.type; + return handle_to_qentry(handle)->s.type; } static odp_schedule_sync_t queue_sched_type(odp_queue_t handle) { - return qentry_from_int(queue_from_ext(handle))->s.param.sched.sync; + return handle_to_qentry(handle)->s.param.sched.sync; } static odp_schedule_prio_t queue_sched_prio(odp_queue_t handle) { - return qentry_from_int(queue_from_ext(handle))->s.param.sched.prio; + return handle_to_qentry(handle)->s.param.sched.prio; } static odp_schedule_group_t queue_sched_group(odp_queue_t handle) { - return qentry_from_int(queue_from_ext(handle))->s.param.sched.group; + return handle_to_qentry(handle)->s.param.sched.group; } static int queue_lock_count(odp_queue_t handle) { - queue_entry_t *queue = qentry_from_int(queue_from_ext(handle)); + queue_entry_t *queue = handle_to_qentry(handle); return queue->s.param.sched.sync == ODP_SCHED_SYNC_ORDERED ? (int)queue->s.param.sched.lock_count : -1; @@ -299,7 +257,7 @@ void sched_cb_queue_destroy_finalize(uint32_t queue_index) static int queue_destroy(odp_queue_t handle) { queue_entry_t *queue; - queue = qentry_from_int(queue_from_ext(handle)); + queue = handle_to_qentry(handle); if (handle == ODP_QUEUE_INVALID) return -1; @@ -352,14 +310,14 @@ static int queue_context_set(odp_queue_t handle, void *context, uint32_t len ODP_UNUSED) { odp_mb_full(); - qentry_from_int(queue_from_ext(handle))->s.param.context = context; + handle_to_qentry(handle)->s.param.context = context; odp_mb_full(); return 0; } static void *queue_context(odp_queue_t handle) { - return qentry_from_int(queue_from_ext(handle))->s.param.context; + return handle_to_qentry(handle)->s.param.context; } static odp_queue_t queue_lookup(const char *name) @@ -385,7 +343,7 @@ static odp_queue_t queue_lookup(const char *name) return ODP_QUEUE_INVALID; } -static inline int enq_multi(queue_t handle, odp_buffer_hdr_t *buf_hdr[], +static inline int enq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int num) { int sched = 0; @@ -393,8 +351,8 @@ static inline int enq_multi(queue_t handle, odp_buffer_hdr_t *buf_hdr[], queue_entry_t *queue; odp_buffer_hdr_t *hdr, *tail, *next_hdr; - queue = qentry_from_int(handle); - if (sched_fn->ord_enq_multi(handle, (void **)buf_hdr, num, &ret)) + queue = qentry_from_int(q_int); + if (sched_fn->ord_enq_multi(q_int, (void **)buf_hdr, num, &ret)) return ret; /* Optimize the common case of single enqueue */ @@ -462,17 +420,17 @@ static inline int enq_multi(queue_t handle, odp_buffer_hdr_t *buf_hdr[], return num; /* All events enqueued */ } -static int _queue_enq_multi(queue_t handle, odp_buffer_hdr_t *buf_hdr[], - int num) +static int queue_int_enq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], + int num) { - return enq_multi(handle, buf_hdr, num); + return enq_multi(q_int, buf_hdr, num); } -static int _queue_enq(queue_t handle, odp_buffer_hdr_t *buf_hdr) +static int queue_int_enq(queue_t q_int, odp_buffer_hdr_t *buf_hdr) { int ret; - ret = enq_multi(handle, &buf_hdr, 1); + ret = enq_multi(q_int, &buf_hdr, 1); if (ret == 1) return 0; @@ -489,7 +447,7 @@ static int queue_enq_multi(odp_queue_t handle, const odp_event_t ev[], int num) if (num > QUEUE_MULTI_MAX) num = QUEUE_MULTI_MAX; - queue = qentry_from_int(queue_from_ext(handle)); + queue = handle_to_qentry(handle); for (i = 0; i < num; i++) buf_hdr[i] = buf_hdl_to_hdr(odp_buffer_from_event(ev[i])); @@ -503,13 +461,13 @@ static int queue_enq(odp_queue_t handle, odp_event_t ev) odp_buffer_hdr_t *buf_hdr; queue_entry_t *queue; - queue = qentry_from_int(queue_from_ext(handle)); + queue = handle_to_qentry(handle); buf_hdr = buf_hdl_to_hdr(odp_buffer_from_event(ev)); return queue->s.enqueue(qentry_to_int(queue), buf_hdr); } -static inline int deq_multi(queue_t handle, odp_buffer_hdr_t *buf_hdr[], +static inline int deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int num) { odp_buffer_hdr_t *hdr, *next; @@ -517,7 +475,7 @@ static inline int deq_multi(queue_t handle, odp_buffer_hdr_t *buf_hdr[], queue_entry_t *queue; int updated = 0; - queue = qentry_from_int(handle); + queue = qentry_from_int(q_int); LOCK(&queue->s.lock); if (odp_unlikely(queue->s.status < QUEUE_STATUS_READY)) { /* Bad queue, or queue has been destroyed. @@ -583,18 +541,18 @@ static inline int deq_multi(queue_t handle, odp_buffer_hdr_t *buf_hdr[], return i; } -static int _queue_deq_multi(queue_t handle, odp_buffer_hdr_t *buf_hdr[], - int num) +static int queue_int_deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], + int num) { - return deq_multi(handle, buf_hdr, num); + return deq_multi(q_int, buf_hdr, num); } -static odp_buffer_hdr_t *_queue_deq(queue_t handle) +static odp_buffer_hdr_t *queue_int_deq(queue_t q_int) { odp_buffer_hdr_t *buf_hdr = NULL; int ret; - ret = deq_multi(handle, &buf_hdr, 1); + ret = deq_multi(q_int, &buf_hdr, 1); if (ret == 1) return buf_hdr; @@ -611,7 +569,7 @@ static int queue_deq_multi(odp_queue_t handle, odp_event_t events[], int num) if (num > QUEUE_MULTI_MAX) num = QUEUE_MULTI_MAX; - queue = qentry_from_int(queue_from_ext(handle)); + queue = handle_to_qentry(handle); ret = queue->s.dequeue_multi(qentry_to_int(queue), buf_hdr, num); @@ -627,7 +585,7 @@ static odp_event_t queue_deq(odp_queue_t handle) queue_entry_t *queue; odp_buffer_hdr_t *buf_hdr; - queue = qentry_from_int(queue_from_ext(handle)); + queue = handle_to_qentry(handle); buf_hdr = queue->s.dequeue(qentry_to_int(queue)); if (buf_hdr) @@ -636,6 +594,49 @@ static odp_event_t queue_deq(odp_queue_t handle) return ODP_EVENT_INVALID; } +static int queue_init(queue_entry_t *queue, const char *name, + const odp_queue_param_t *param) +{ + if (name == NULL) { + queue->s.name[0] = 0; + } else { + strncpy(queue->s.name, name, ODP_QUEUE_NAME_LEN - 1); + queue->s.name[ODP_QUEUE_NAME_LEN - 1] = 0; + } + memcpy(&queue->s.param, param, sizeof(odp_queue_param_t)); + if (queue->s.param.sched.lock_count > sched_fn->max_ordered_locks()) + return -1; + + if (param->type == ODP_QUEUE_TYPE_SCHED) { + queue->s.param.deq_mode = ODP_QUEUE_OP_DISABLED; + + if (param->sched.sync == ODP_SCHED_SYNC_ORDERED) { + unsigned i; + + odp_atomic_init_u64(&queue->s.ordered.ctx, 0); + odp_atomic_init_u64(&queue->s.ordered.next_ctx, 0); + + for (i = 0; i < queue->s.param.sched.lock_count; i++) + odp_atomic_init_u64(&queue->s.ordered.lock[i], + 0); + } + } + queue->s.type = queue->s.param.type; + + queue->s.enqueue = queue_int_enq; + queue->s.dequeue = queue_int_deq; + queue->s.enqueue_multi = queue_int_enq_multi; + queue->s.dequeue_multi = queue_int_deq_multi; + + queue->s.pktin = PKTIN_INVALID; + queue->s.pktout = PKTOUT_INVALID; + + queue->s.head = NULL; + queue->s.tail = NULL; + + return 0; +} + void queue_lock(queue_entry_t *queue) { LOCK(&queue->s.lock); @@ -776,64 +777,65 @@ static uint64_t queue_to_u64(odp_queue_t hdl) return _odp_pri(hdl); } -static odp_pktout_queue_t queue_get_pktout(queue_t handle) +static odp_pktout_queue_t queue_get_pktout(queue_t q_int) { - return qentry_from_int(handle)->s.pktout; + return qentry_from_int(q_int)->s.pktout; } -static void queue_set_pktout(queue_t handle, odp_pktio_t pktio, int index) +static void queue_set_pktout(queue_t q_int, odp_pktio_t pktio, int index) { - qentry_from_int(handle)->s.pktout.pktio = pktio; - qentry_from_int(handle)->s.pktout.index = index; + queue_entry_t *qentry = qentry_from_int(q_int); + + qentry->s.pktout.pktio = pktio; + qentry->s.pktout.index = index; } -static odp_pktin_queue_t queue_get_pktin(queue_t handle) +static odp_pktin_queue_t queue_get_pktin(queue_t q_int) { - return qentry_from_int(handle)->s.pktin; + return qentry_from_int(q_int)->s.pktin; } -static void queue_set_pktin(queue_t handle, odp_pktio_t pktio, int index) +static void queue_set_pktin(queue_t q_int, odp_pktio_t pktio, int index) { - qentry_from_int(handle)->s.pktin.pktio = pktio; - qentry_from_int(handle)->s.pktin.index = index; + queue_entry_t *qentry = qentry_from_int(q_int); + + qentry->s.pktin.pktio = pktio; + qentry->s.pktin.index = index; } -static void queue_set_enq_func(queue_t handle, queue_enq_fn_t func) +static void queue_set_enq_func(queue_t q_int, queue_enq_fn_t func) { - qentry_from_int(handle)->s.enqueue = func; + qentry_from_int(q_int)->s.enqueue = func; } -static void queue_set_enq_multi_func(queue_t handle, queue_enq_multi_fn_t func) +static void queue_set_enq_multi_func(queue_t q_int, queue_enq_multi_fn_t func) { - qentry_from_int(handle)->s.enqueue_multi = func; + qentry_from_int(q_int)->s.enqueue_multi = func; } -static void queue_set_deq_func(queue_t handle, queue_deq_fn_t func) +static void queue_set_deq_func(queue_t q_int, queue_deq_fn_t func) { - qentry_from_int(handle)->s.dequeue = func; + qentry_from_int(q_int)->s.dequeue = func; } -static void queue_set_deq_multi_func(queue_t handle, queue_deq_multi_fn_t func) +static void queue_set_deq_multi_func(queue_t q_int, queue_deq_multi_fn_t func) { - qentry_from_int(handle)->s.dequeue_multi = func; + qentry_from_int(q_int)->s.dequeue_multi = func; } -static void queue_set_type(queue_t handle, odp_queue_type_t type) +static void queue_set_type(queue_t q_int, odp_queue_type_t type) { - qentry_from_int(handle)->s.type = type; + qentry_from_int(q_int)->s.type = type; } static queue_t queue_from_ext(odp_queue_t handle) { - uint32_t queue_id; - - queue_id = queue_to_id(handle); - return qentry_to_int(get_qentry(queue_id)); + return qentry_to_int(handle_to_qentry(handle)); } -static odp_queue_t queue_to_ext(queue_t handle) +static odp_queue_t queue_to_ext(queue_t q_int) { - return qentry_from_int(handle)->s.handle; + return qentry_from_int(q_int)->s.handle; } /* API functions */ @@ -866,10 +868,10 @@ queue_fn_t queue_default_fn = { .term_local = queue_term_local, .from_ext = queue_from_ext, .to_ext = queue_to_ext, - .enq = _queue_enq, - .enq_multi = _queue_enq_multi, - .deq = _queue_deq, - .deq_multi = _queue_deq_multi, + .enq = queue_int_enq, + .enq_multi = queue_int_enq_multi, + .deq = queue_int_deq, + .deq_multi = queue_int_deq_multi, .get_pktout = queue_get_pktout, .set_pktout = queue_set_pktout, .get_pktin = queue_get_pktin, diff --git a/platform/linux-generic/odp_schedule.c b/platform/linux-generic/odp_schedule.c index 011d4dc4..92634235 100644 --- a/platform/linux-generic/odp_schedule.c +++ b/platform/linux-generic/odp_schedule.c @@ -712,12 +712,12 @@ static inline int copy_events(odp_event_t out_ev[], unsigned int max) return i; } -static int schedule_ord_enq_multi(queue_t handle, void *buf_hdr[], +static int schedule_ord_enq_multi(queue_t q_int, void *buf_hdr[], int num, int *ret) { int i; uint32_t stash_num = sched_local.ordered.stash_num; - queue_entry_t *dst_queue = qentry_from_int(handle); + queue_entry_t *dst_queue = qentry_from_int(q_int); queue_entry_t *src_queue = sched_local.ordered.src_queue; if (!sched_local.ordered.src_queue || sched_local.ordered.in_order) diff --git a/platform/linux-generic/odp_schedule_iquery.c b/platform/linux-generic/odp_schedule_iquery.c index bdf1a460..79086843 100644 --- a/platform/linux-generic/odp_schedule_iquery.c +++ b/platform/linux-generic/odp_schedule_iquery.c @@ -1159,12 +1159,12 @@ static inline void schedule_release_context(void) schedule_release_atomic(); } -static int schedule_ord_enq_multi(queue_t handle, void *buf_hdr[], +static int schedule_ord_enq_multi(queue_t q_int, void *buf_hdr[], int num, int *ret) { int i; uint32_t stash_num = thread_local.ordered.stash_num; - queue_entry_t *dst_queue = qentry_from_int(handle); + queue_entry_t *dst_queue = qentry_from_int(q_int); queue_entry_t *src_queue = thread_local.ordered.src_queue; if (!thread_local.ordered.src_queue || thread_local.ordered.in_order) diff --git a/platform/linux-generic/odp_schedule_sp.c b/platform/linux-generic/odp_schedule_sp.c index 91d70e3a..66715c13 100644 --- a/platform/linux-generic/odp_schedule_sp.c +++ b/platform/linux-generic/odp_schedule_sp.c @@ -414,10 +414,10 @@ static int unsched_queue(uint32_t qi ODP_UNUSED) return 0; } -static int ord_enq_multi(queue_t handle, void *buf_hdr[], int num, +static int ord_enq_multi(queue_t q_int, void *buf_hdr[], int num, int *ret) { - (void)handle; + (void)q_int; (void)buf_hdr; (void)num; (void)ret; commit 200782c3da2104a020f278691b3b23b8d55893e0 Author: Honnappa Nagarahalli <honnappa.nagarahalli(a)arm.com> Date: Fri Jun 2 13:27:00 2017 -0500 linux-generic: queue: modular queue interface Created abstract queue type. Queue APIs and functions towards the internal components are converted into function pointers. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli(a)arm.com> diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index 989a65b6..45dd0f4d 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -156,6 +156,7 @@ noinst_HEADERS = \ ${srcdir}/include/odp_posix_extensions.h \ ${srcdir}/include/odp_queue_internal.h \ ${srcdir}/include/odp_ring_internal.h \ + ${srcdir}/include/odp_queue_if.h \ ${srcdir}/include/odp_schedule_if.h \ ${srcdir}/include/odp_sorted_list_internal.h \ ${srcdir}/include/odp_shm_internal.h \ @@ -208,6 +209,7 @@ __LIB__libodp_linux_la_SOURCES = \ odp_pkt_queue.c \ odp_pool.c \ odp_queue.c \ + odp_queue_if.c \ odp_rwlock.c \ odp_rwlock_recursive.c \ odp_schedule.c \ diff --git a/platform/linux-generic/include/odp_classification_datamodel.h b/platform/linux-generic/include/odp_classification_datamodel.h index 9df756bf..fbe10cb4 100644 --- a/platform/linux-generic/include/odp_classification_datamodel.h +++ b/platform/linux-generic/include/odp_classification_datamodel.h @@ -24,7 +24,7 @@ extern "C" { #include <odp_pool_internal.h> #include <odp_packet_internal.h> #include <odp_packet_io_internal.h> -#include <odp_queue_internal.h> +#include <odp_queue_if.h> #include <protocols/ip.h> /* Maximum Class Of Service Entry */ @@ -85,7 +85,7 @@ typedef struct pmr_term_value { Class Of Service */ struct cos_s { - queue_entry_t *queue; /* Associated Queue */ + queue_t queue; /* Associated Queue */ odp_pool_t pool; /* Associated Buffer pool */ union pmr_u *pmr[ODP_PMR_PER_COS_MAX]; /* Chained PMR */ union cos_u *linked_cos[ODP_PMR_PER_COS_MAX]; /* Chained CoS with PMR*/ diff --git a/platform/linux-generic/include/odp_packet_io_queue.h b/platform/linux-generic/include/odp_packet_io_queue.h index d1d4b225..71333ddc 100644 --- a/platform/linux-generic/include/odp_packet_io_queue.h +++ b/platform/linux-generic/include/odp_packet_io_queue.h @@ -18,7 +18,7 @@ extern "C" { #endif -#include <odp_queue_internal.h> +#include <odp_queue_if.h> #include <odp_buffer_internal.h> #include <odp_config_internal.h> @@ -28,20 +28,18 @@ extern "C" { ODP_STATIC_ASSERT(ODP_PKTIN_QUEUE_MAX_BURST >= QUEUE_MULTI_MAX, "ODP_PKTIN_DEQ_MULTI_MAX_ERROR"); -int pktin_enqueue(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr); -odp_buffer_hdr_t *pktin_dequeue(queue_entry_t *queue); +int pktin_enqueue(queue_t queue, odp_buffer_hdr_t *buf_hdr); +odp_buffer_hdr_t *pktin_dequeue(queue_t queue); -int pktin_enq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], int num); -int pktin_deq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], int num); +int pktin_enq_multi(queue_t queue, odp_buffer_hdr_t *buf_hdr[], int num); +int pktin_deq_multi(queue_t queue, odp_buffer_hdr_t *buf_hdr[], int num); -int pktout_enqueue(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr); -odp_buffer_hdr_t *pktout_dequeue(queue_entry_t *queue); +int pktout_enqueue(queue_t queue, odp_buffer_hdr_t *buf_hdr); +odp_buffer_hdr_t *pktout_dequeue(queue_t queue); -int pktout_enq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], - int num); -int pktout_deq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], - int num); +int pktout_enq_multi(queue_t queue, odp_buffer_hdr_t *buf_hdr[], int num); +int pktout_deq_multi(queue_t queue, odp_buffer_hdr_t *buf_hdr[], int num); #ifdef __cplusplus } diff --git a/platform/linux-generic/include/odp_queue_if.h b/platform/linux-generic/include/odp_queue_if.h new file mode 100644 index 00000000..4359435f --- /dev/null +++ b/platform/linux-generic/include/odp_queue_if.h @@ -0,0 +1,104 @@ +/* Copyright (c) 2017, ARM Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef ODP_QUEUE_IF_H_ +#define ODP_QUEUE_IF_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <odp/api/queue.h> +#include <odp/api/schedule.h> +#include <odp/api/packet_io.h> +#include <odp_forward_typedefs_internal.h> + +#define QUEUE_MULTI_MAX CONFIG_BURST_SIZE + +/* Queue API functions */ +typedef struct { + odp_queue_t (*queue_create)(const char *name, + const odp_queue_param_t *param); + int (*queue_destroy)(odp_queue_t queue); + odp_queue_t (*queue_lookup)(const char *name); + int (*queue_capability)(odp_queue_capability_t *capa); + int (*queue_context_set)(odp_queue_t queue, void *context, + uint32_t len); + void *(*queue_context)(odp_queue_t queue); + int (*queue_enq)(odp_queue_t queue, odp_event_t ev); + int (*queue_enq_multi)(odp_queue_t queue, const odp_event_t events[], + int num); + odp_event_t (*queue_deq)(odp_queue_t queue); + int (*queue_deq_multi)(odp_queue_t queue, odp_event_t events[], + int num); + odp_queue_type_t (*queue_type)(odp_queue_t queue); + odp_schedule_sync_t (*queue_sched_type)(odp_queue_t queue); + odp_schedule_prio_t (*queue_sched_prio)(odp_queue_t queue); + odp_schedule_group_t (*queue_sched_group)(odp_queue_t queue); + int (*queue_lock_count)(odp_queue_t queue); + uint64_t (*queue_to_u64)(odp_queue_t hdl); + void (*queue_param_init)(odp_queue_param_t *param); + int (*queue_info)(odp_queue_t queue, odp_queue_info_t *info); +} queue_api_t; + +/* Internal abstract queue handle */ +typedef struct { char dummy; } _queue_t; +typedef _queue_t *queue_t; + +typedef int (*queue_init_global_fn_t)(void); +typedef int (*queue_term_global_fn_t)(void); +typedef int (*queue_init_local_fn_t)(void); +typedef int (*queue_term_local_fn_t)(void); +typedef queue_t (*queue_from_ext_fn_t)(odp_queue_t handle); +typedef odp_queue_t (*queue_to_ext_fn_t)(queue_t handle); +typedef int (*queue_enq_fn_t)(queue_t handle, odp_buffer_hdr_t *); +typedef int (*queue_enq_multi_fn_t)(queue_t handle, odp_buffer_hdr_t **, int); +typedef odp_buffer_hdr_t *(*queue_deq_fn_t)(queue_t handle); +typedef int (*queue_deq_multi_fn_t)(queue_t handle, odp_buffer_hdr_t **, int); +typedef odp_pktout_queue_t (*queue_get_pktout_fn_t)(queue_t handle); +typedef void (*queue_set_pktout_fn_t)(queue_t handle, odp_pktio_t pktio, + int index); +typedef odp_pktin_queue_t (*queue_get_pktin_fn_t)(queue_t handle); +typedef void (*queue_set_pktin_fn_t)(queue_t handle, odp_pktio_t pktio, + int index); +typedef void (*queue_set_enq_fn_t)(queue_t handle, queue_enq_fn_t func); +typedef void (*queue_set_enq_multi_fn_t)(queue_t handle, + queue_enq_multi_fn_t func); +typedef void (*queue_set_deq_fn_t)(queue_t handle, queue_deq_fn_t func); +typedef void (*queue_set_deq_multi_fn_t)(queue_t handle, + queue_deq_multi_fn_t func); +typedef void (*queue_set_type_fn_t)(queue_t handle, odp_queue_type_t type); + +/* Queue functions towards other internal components */ +typedef struct { + queue_init_global_fn_t init_global; + queue_term_global_fn_t term_global; + queue_init_local_fn_t init_local; + queue_term_local_fn_t term_local; + queue_from_ext_fn_t from_ext; + queue_to_ext_fn_t to_ext; + queue_enq_fn_t enq; + queue_enq_multi_fn_t enq_multi; + queue_deq_fn_t deq; + queue_deq_multi_fn_t deq_multi; + queue_get_pktout_fn_t get_pktout; + queue_set_pktout_fn_t set_pktout; + queue_get_pktin_fn_t get_pktin; + queue_set_pktin_fn_t set_pktin; + queue_set_enq_fn_t set_enq_fn; + queue_set_enq_multi_fn_t set_enq_multi_fn; + queue_set_deq_fn_t set_deq_fn; + queue_set_deq_multi_fn_t set_deq_multi_fn; + queue_set_type_fn_t set_type; +} queue_fn_t; + +extern const queue_fn_t *queue_fn; + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/platform/linux-generic/include/odp_queue_internal.h b/platform/linux-generic/include/odp_queue_internal.h index 560f826e..0c31ce8a 100644 --- a/platform/linux-generic/include/odp_queue_internal.h +++ b/platform/linux-generic/include/odp_queue_internal.h @@ -20,7 +20,7 @@ extern "C" { #include <odp/api/queue.h> #include <odp_forward_typedefs_internal.h> -#include <odp_schedule_if.h> +#include <odp_queue_if.h> #include <odp_buffer_internal.h> #include <odp_align_internal.h> #include <odp/api/packet_io.h> @@ -29,26 +29,12 @@ extern "C" { #include <odp/api/ticketlock.h> #include <odp_config_internal.h> -#define QUEUE_MULTI_MAX CONFIG_BURST_SIZE - #define QUEUE_STATUS_FREE 0 #define QUEUE_STATUS_DESTROYED 1 #define QUEUE_STATUS_READY 2 #define QUEUE_STATUS_NOTSCHED 3 #define QUEUE_STATUS_SCHED 4 - -/* forward declaration */ -union queue_entry_u; - -typedef int (*enq_func_t)(union queue_entry_u *, odp_buffer_hdr_t *); -typedef odp_buffer_hdr_t *(*deq_func_t)(union queue_entry_u *); - -typedef int (*enq_multi_func_t)(union queue_entry_u *, - odp_buffer_hdr_t **, int); -typedef int (*deq_multi_func_t)(union queue_entry_u *, - odp_buffer_hdr_t **, int); - struct queue_entry_s { odp_ticketlock_t lock ODP_ALIGNED_CACHE; @@ -63,10 +49,10 @@ struct queue_entry_s { odp_atomic_u64_t lock[CONFIG_QUEUE_MAX_ORD_LOCKS]; } ordered ODP_ALIGNED_CACHE; - enq_func_t enqueue ODP_ALIGNED_CACHE; - deq_func_t dequeue; - enq_multi_func_t enqueue_multi; - deq_multi_func_t dequeue_multi; + queue_enq_fn_t enqueue ODP_ALIGNED_CACHE; + queue_deq_fn_t dequeue; + queue_enq_multi_fn_t enqueue_multi; + queue_deq_multi_fn_t dequeue_multi; uint32_t index; odp_queue_t handle; @@ -82,15 +68,8 @@ union queue_entry_u { uint8_t pad[ROUNDUP_CACHE_LINE(sizeof(struct queue_entry_s))]; }; - queue_entry_t *get_qentry(uint32_t queue_id); -int queue_enq(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr); -odp_buffer_hdr_t *queue_deq(queue_entry_t *queue); - -int queue_enq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], int num); -int queue_deq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], int num); - void queue_lock(queue_entry_t *queue); void queue_unlock(queue_entry_t *queue); @@ -99,12 +78,14 @@ static inline uint32_t queue_to_id(odp_queue_t handle) return _odp_typeval(handle) - 1; } -static inline queue_entry_t *queue_to_qentry(odp_queue_t handle) +static inline queue_entry_t *qentry_from_int(queue_t handle) { - uint32_t queue_id; + return (queue_entry_t *)(void *)(handle); +} - queue_id = queue_to_id(handle); - return get_qentry(queue_id); +static inline queue_t qentry_to_int(queue_entry_t *qentry) +{ + return (queue_t)(qentry); } #ifdef __cplusplus diff --git a/platform/linux-generic/include/odp_schedule_if.h b/platform/linux-generic/include/odp_schedule_if.h index 530d157f..9adacef7 100644 --- a/platform/linux-generic/include/odp_schedule_if.h +++ b/platform/linux-generic/include/odp_schedule_if.h @@ -26,7 +26,7 @@ typedef int (*schedule_init_queue_fn_t)(uint32_t queue_index, typedef void (*schedule_destroy_queue_fn_t)(uint32_t queue_index); typedef int (*schedule_sched_queue_fn_t)(uint32_t queue_index); typedef int (*schedule_unsched_queue_fn_t)(uint32_t queue_index); -typedef int (*schedule_ord_enq_multi_fn_t)(uint32_t queue_index, +typedef int (*schedule_ord_enq_multi_fn_t)(queue_t handle, void *buf_hdr[], int num, int *ret); typedef int (*schedule_init_global_fn_t)(void); typedef int (*schedule_term_global_fn_t)(void); diff --git a/platform/linux-generic/include/odp_traffic_mngr_internal.h b/platform/linux-generic/include/odp_traffic_mngr_internal.h index 9f821fe4..e8254f5e 100644 --- a/platform/linux-generic/include/odp_traffic_mngr_internal.h +++ b/platform/linux-generic/include/odp_traffic_mngr_internal.h @@ -29,7 +29,7 @@ extern "C" { #include <odp_internal.h> #include <odp_debug_internal.h> #include <odp_buffer_internal.h> -#include <odp_queue_internal.h> +#include <odp_queue_if.h> #include <odp_packet_internal.h> typedef struct stat file_stat_t; @@ -286,7 +286,7 @@ struct tm_queue_obj_s { uint8_t tm_idx; uint8_t delayed_cnt; uint8_t blocked_cnt; - queue_entry_t tm_qentry; + queue_t tm_qentry; }; struct tm_node_obj_s { diff --git a/platform/linux-generic/odp_classification.c b/platform/linux-generic/odp_classification.c index 7ebc47d7..5f153e73 100644 --- a/platform/linux-generic/odp_classification.c +++ b/platform/linux-generic/odp_classification.c @@ -157,14 +157,14 @@ int odp_cls_capability(odp_cls_capability_t *capability) odp_cos_t odp_cls_cos_create(const char *name, odp_cls_cos_param_t *param) { int i, j; - queue_entry_t *queue; + queue_t queue; odp_cls_drop_t drop_policy; /* Packets are dropped if Queue or Pool is invalid*/ if (param->queue == ODP_QUEUE_INVALID) queue = NULL; else - queue = queue_to_qentry(param->queue); + queue = queue_fn->from_ext(param->queue); drop_policy = param->drop_policy; @@ -266,7 +266,7 @@ int odp_cos_queue_set(odp_cos_t cos_id, odp_queue_t queue_id) if (queue_id == ODP_QUEUE_INVALID) cos->s.queue = NULL; else - cos->s.queue = queue_to_qentry(queue_id); + cos->s.queue = queue_fn->from_ext(queue_id); return 0; } @@ -282,7 +282,7 @@ odp_queue_t odp_cos_queue(odp_cos_t cos_id) if (!cos->s.queue) return ODP_QUEUE_INVALID; - return cos->s.queue->s.handle; + return queue_fn->to_ext(cos->s.queue); } int odp_cos_drop_set(odp_cos_t cos_id, odp_cls_drop_t drop_policy) @@ -846,7 +846,7 @@ int cls_classify_packet(pktio_entry_t *entry, const uint8_t *base, *pool = cos->s.pool; pkt_hdr->p.input_flags.dst_queue = 1; - pkt_hdr->dst_queue = cos->s.queue->s.handle; + pkt_hdr->dst_queue = queue_fn->to_ext(cos->s.queue); return 0; } diff --git a/platform/linux-generic/odp_init.c b/platform/linux-generic/odp_init.c index ab2d6fb0..35a285ea 100644 --- a/platform/linux-generic/odp_init.c +++ b/platform/linux-generic/odp_init.c @@ -83,7 +83,7 @@ int odp_init_global(odp_instance_t *instance, } stage = POOL_INIT; - if (odp_queue_init_global()) { + if (queue_fn->init_global()) { ODP_ERR("ODP queue init failed.\n"); goto init_failed; } @@ -204,7 +204,7 @@ int _odp_term_global(enum init_stage stage) /* Fall through */ case QUEUE_INIT: - if (odp_queue_term_global()) { + if (queue_fn->term_global()) { ODP_ERR("ODP queue term failed.\n"); rc = -1; } @@ -299,6 +299,12 @@ int odp_init_local(odp_instance_t instance, odp_thread_type_t thr_type) } stage = POOL_INIT; + if (queue_fn->init_local()) { + ODP_ERR("ODP queue local init failed.\n"); + goto init_fail; + } + stage = QUEUE_INIT; + if (sched_fn->init_local()) { ODP_ERR("ODP schedule local init failed.\n"); goto init_fail; @@ -332,6 +338,13 @@ int _odp_term_local(enum init_stage stage) } /* Fall through */ + case QUEUE_INIT: + if (queue_fn->term_local()) { + ODP_ERR("ODP queue local term failed.\n"); + rc = -1; + } + /* Fall through */ + case POOL_INIT: if (odp_pool_term_local()) { ODP_ERR("ODP buffer pool local term failed.\n"); diff --git a/platform/linux-generic/odp_packet_io.c b/platform/linux-generic/odp_packet_io.c index e5436a90..4dd28549 100644 --- a/platform/linux-generic/odp_packet_io.c +++ b/platform/linux-generic/odp_packet_io.c @@ -16,7 +16,7 @@ #include <odp/api/shared_memory.h> #include <odp_packet_socket.h> #include <odp_config_internal.h> -#include <odp_queue_internal.h> +#include <odp_queue_if.h> #include <odp_schedule_if.h> #include <odp_classification_internal.h> #include <odp_debug_internal.h> @@ -566,11 +566,11 @@ static inline int pktin_recv_buf(odp_pktin_queue_t queue, buf_hdr = packet_to_buf_hdr(pkt); if (pkt_hdr->p.input_flags.dst_queue) { - queue_entry_t *dst_queue; + queue_t dst_queue; int ret; - dst_queue = queue_to_qentry(pkt_hdr->dst_queue); - ret = queue_enq(dst_queue, buf_hdr); + dst_queue = queue_fn->from_ext(pkt_hdr->dst_queue); + ret = queue_fn->enq(dst_queue, buf_hdr); if (ret < 0) odp_packet_free(pkt); continue; @@ -580,45 +580,42 @@ static inline int pktin_recv_buf(odp_pktin_queue_t queue, return num_rx; } -int pktout_enqueue(queue_entry_t *qentry, odp_buffer_hdr_t *buf_hdr) +int pktout_enqueue(queue_t qentry, odp_buffer_hdr_t *buf_hdr) { odp_packet_t pkt = packet_from_buf_hdr(buf_hdr); int len = 1; int nbr; - if (sched_fn->ord_enq_multi(qentry->s.index, (void **)buf_hdr, len, - &nbr)) + if (sched_fn->ord_enq_multi(qentry, (void **)buf_hdr, len, &nbr)) return (nbr == len ? 0 : -1); - nbr = odp_pktout_send(qentry->s.pktout, &pkt, len); + nbr = odp_pktout_send(queue_fn->get_pktout(qentry), &pkt, len); return (nbr == len ? 0 : -1); } -odp_buffer_hdr_t *pktout_dequeue(queue_entry_t *qentry ODP_UNUSED) +odp_buffer_hdr_t *pktout_dequeue(queue_t qentry ODP_UNUSED) { ODP_ABORT("attempted dequeue from a pktout queue"); return NULL; } -int pktout_enq_multi(queue_entry_t *qentry, odp_buffer_hdr_t *buf_hdr[], - int num) +int pktout_enq_multi(queue_t qentry, odp_buffer_hdr_t *buf_hdr[], int num) { odp_packet_t pkt_tbl[QUEUE_MULTI_MAX]; int nbr; int i; - if (sched_fn->ord_enq_multi(qentry->s.index, (void **)buf_hdr, num, - &nbr)) + if (sched_fn->ord_enq_multi(qentry, (void **)buf_hdr, num, &nbr)) return nbr; for (i = 0; i < num; ++i) pkt_tbl[i] = packet_from_buf_hdr(buf_hdr[i]); - nbr = odp_pktout_send(qentry->s.pktout, pkt_tbl, num); + nbr = odp_pktout_send(queue_fn->get_pktout(qentry), pkt_tbl, num); return nbr; } -int pktout_deq_multi(queue_entry_t *qentry ODP_UNUSED, +int pktout_deq_multi(queue_t qentry ODP_UNUSED, odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, int num ODP_UNUSED) { @@ -626,48 +623,49 @@ int pktout_deq_multi(queue_entry_t *qentry ODP_UNUSED, return 0; } -int pktin_enqueue(queue_entry_t *qentry ODP_UNUSED, +int pktin_enqueue(queue_t qentry ODP_UNUSED, odp_buffer_hdr_t *buf_hdr ODP_UNUSED) { ODP_ABORT("attempted enqueue to a pktin queue"); return -1; } -odp_buffer_hdr_t *pktin_dequeue(queue_entry_t *qentry) +odp_buffer_hdr_t *pktin_dequeue(queue_t qentry) { odp_buffer_hdr_t *buf_hdr; odp_buffer_hdr_t *hdr_tbl[QUEUE_MULTI_MAX]; int pkts; - buf_hdr = queue_deq(qentry); + buf_hdr = queue_fn->deq(qentry); if (buf_hdr != NULL) return buf_hdr; - pkts = pktin_recv_buf(qentry->s.pktin, hdr_tbl, QUEUE_MULTI_MAX); + pkts = pktin_recv_buf(queue_fn->get_pktin(qentry), + hdr_tbl, QUEUE_MULTI_MAX); if (pkts <= 0) return NULL; if (pkts > 1) - queue_enq_multi(qentry, &hdr_tbl[1], pkts - 1); + queue_fn->enq_multi(qentry, &hdr_tbl[1], pkts - 1); buf_hdr = hdr_tbl[0]; return buf_hdr; } -int pktin_enq_multi(queue_entry_t *qentry ODP_UNUSED, +int pktin_enq_multi(queue_t qentry ODP_UNUSED, odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, int num ODP_UNUSED) { ODP_ABORT("attempted enqueue to a pktin queue"); return 0; } -int pktin_deq_multi(queue_entry_t *qentry, odp_buffer_hdr_t *buf_hdr[], int num) +int pktin_deq_multi(queue_t qentry, odp_buffer_hdr_t *buf_hdr[], int num) { int nbr; odp_buffer_hdr_t *hdr_tbl[QUEUE_MULTI_MAX]; int pkts, i, j; - nbr = queue_deq_multi(qentry, buf_hdr, num); + nbr = queue_fn->deq_multi(qentry, buf_hdr, num); if (odp_unlikely(nbr > num)) ODP_ABORT("queue_deq_multi req: %d, returned %d\n", num, nbr); @@ -677,7 +675,8 @@ int pktin_deq_multi(queue_entry_t *qentry, odp_buffer_hdr_t *buf_hdr[], int num) if (nbr == num) return nbr; - pkts = pktin_recv_buf(qentry->s.pktin, hdr_tbl, QUEUE_MULTI_MAX); + pkts = pktin_recv_buf(queue_fn->get_pktin(qentry), + hdr_tbl, QUEUE_MULTI_MAX); if (pkts <= 0) return nbr; @@ -689,7 +688,7 @@ int pktin_deq_multi(queue_entry_t *qentry, odp_buffer_hdr_t *buf_hdr[], int num) hdr_tbl[j] = hdr_tbl[i]; if (j) - queue_enq_multi(qentry, hdr_tbl, j); + queue_fn->enq_multi(qentry, hdr_tbl, j); return nbr; } @@ -711,7 +710,7 @@ int sched_cb_pktin_poll(int pktio_index, int num_queue, int index[]) } for (idx = 0; idx < num_queue; idx++) { - queue_entry_t *qentry; + queue_t qentry; odp_queue_t queue; odp_pktin_queue_t pktin = entry->s.in_queue[index[idx]].pktin; @@ -726,8 +725,8 @@ int sched_cb_pktin_poll(int pktio_index, int num_queue, int index[]) } queue = entry->s.in_queue[index[idx]].queue; - qentry = queue_to_qentry(queue); - queue_enq_multi(qentry, hdr_tbl, num); + qentry = queue_fn->from_ext(queue); + queue_fn->enq_multi(qentry, hdr_tbl, num); } return 0; @@ -1271,16 +1270,16 @@ int odp_pktin_queue_config(odp_pktio_t pktio, } if (mode == ODP_PKTIN_MODE_QUEUE) { - queue_entry_t *qentry; - - qentry = queue_to_qentry(queue); - qentry->s.pktin.index = i; - qentry->s.pktin.pktio = pktio; - - qentry->s.enqueue = pktin_enqueue; - qentry->s.dequeue = pktin_dequeue; - qentry->s.enqueue_multi = pktin_enq_multi; - qentry->s.dequeue_multi = pktin_deq_multi; + queue_t qentry; + + qentry = queue_fn->from_ext(queue); + queue_fn->set_pktin(qentry, pktio, i); + queue_fn->set_enq_fn(qentry, pktin_enqueue); + queue_fn->set_deq_fn(qentry, pktin_dequeue); + queue_fn->set_enq_multi_fn(qentry, + pktin_enq_multi); + queue_fn->set_deq_multi_fn(qentry, + pktin_deq_multi); } entry->s.in_queue[i].queue = queue; @@ -1377,7 +1376,7 @@ int odp_pktout_queue_config(odp_pktio_t pktio, for (i = 0; i < num_queues; i++) { odp_queue_t queue; odp_queue_param_t queue_param; - queue_entry_t *qentry; + queue_t qentry; char name[ODP_QUEUE_NAME_LEN]; int pktio_id = pktio_to_id(pktio); @@ -1397,15 +1396,14 @@ int odp_pktout_queue_config(odp_pktio_t pktio, return -1; } - qentry = queue_to_qentry(queue); - qentry->s.pktout.index = i; - qentry->s.pktout.pktio = pktio; + qentry = queue_fn->from_ext(queue); + queue_fn->set_pktout(qentry, pktio, i); /* Override default enqueue / dequeue functions */ - qentry->s.enqueue = pktout_enqueue; - qentry->s.dequeue = pktout_dequeue; - qentry->s.enqueue_multi = pktout_enq_multi; - qentry->s.dequeue_multi = pktout_deq_multi; + queue_fn->set_enq_fn(qentry, pktout_enqueue); + queue_fn->set_deq_fn(qentry, pktout_dequeue); + queue_fn->set_enq_multi_fn(qentry, pktout_enq_multi); + queue_fn->set_deq_multi_fn(qentry, pktout_deq_multi); entry->s.out_queue[i].queue = queue; } diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c index dd430cd1..3e18f578 100644 --- a/platform/linux-generic/odp_queue.c +++ b/platform/linux-generic/odp_queue.c @@ -6,6 +6,7 @@ #include <odp/api/queue.h> #include <odp_queue_internal.h> +#include <odp_queue_if.h> #include <odp/api/std_types.h> #include <odp/api/align.h> #include <odp/api/buffer.h> @@ -40,6 +41,15 @@ typedef struct queue_table_t { static queue_table_t *queue_tbl; +static queue_t queue_from_ext(odp_queue_t handle); +static int _queue_enq(queue_t handle, odp_buffer_hdr_t *buf_hdr); +static odp_buffer_hdr_t *_queue_deq(queue_t handle); + +static int _queue_enq_multi(queue_t handle, odp_buffer_hdr_t *buf_hdr[], + int num); +static int _queue_deq_multi(queue_t handle, odp_buffer_hdr_t *buf_hdr[], + int num); + static inline odp_queue_t queue_from_id(uint32_t queue_id) { return _odp_cast_scalar(odp_queue_t, queue_id + 1); @@ -89,10 +99,10 @@ static int queue_init(queue_entry_t *queue, const char *name, } queue->s.type = queue->s.param.type; - queue->s.enqueue = queue_enq; - queue->s.dequeue = queue_deq; - queue->s.enqueue_multi = queue_enq_multi; - queue->s.dequeue_multi = queue_deq_multi; + queue->s.enqueue = _queue_enq; + queue->s.dequeue = _queue_deq; + queue->s.enqueue_multi = _queue_enq_multi; + queue->s.dequeue_multi = _queue_deq_multi; queue->s.pktin = PKTIN_INVALID; queue->s.pktout = PKTOUT_INVALID; @@ -104,7 +114,7 @@ static int queue_init(queue_entry_t *queue, const char *name, } -int odp_queue_init_global(void) +static int queue_init_global(void) { uint32_t i; odp_shm_t shm; @@ -141,7 +151,17 @@ int odp_queue_init_global(void) return 0; } -int odp_queue_term_global(void) +static int queue_init_local(void) +{ + return 0; +} + +static int queue_term_local(void) +{ + return 0; +} + +static int queue_term_global(void) { int ret = 0; int rc = 0; @@ -167,7 +187,7 @@ int odp_queue_term_global(void) return rc; } -int odp_queue_capability(odp_queue_capability_t *capa) +static int queue_capability(odp_queue_capability_t *capa) { memset(capa, 0, sizeof(odp_queue_capability_t)); @@ -182,51 +202,36 @@ int odp_queue_capability(odp_queue_capability_t *capa) return 0; } -odp_queue_type_t odp_queue_type(odp_queue_t handle) +static odp_queue_type_t queue_type(odp_queue_t handle) { - queue_entry_t *queue; - - queue = queue_to_qentry(handle); - - return queue->s.type; + return qentry_from_int(queue_from_ext(handle))->s.type; } -odp_schedule_sync_t odp_queue_sched_type(odp_queue_t handle) +static odp_schedule_sync_t queue_sched_type(odp_queue_t handle) { - queue_entry_t *queue; - - queue = queue_to_qentry(handle); - - return queue->s.param.sched.sync; + return qentry_from_int(queue_from_ext(handle))->s.param.sched.sync; } -odp_schedule_prio_t odp_queue_sched_prio(odp_queue_t handle) +static odp_schedule_prio_t queue_sched_prio(odp_queue_t handle) { - queue_entry_t *queue; - - queue = queue_to_qentry(handle); - - return queue->s.param.sched.prio; + return qentry_from_int(queue_from_ext(handle))->s.param.sched.prio; } -odp_schedule_group_t odp_queue_sched_group(odp_queue_t handle) +static odp_schedule_group_t queue_sched_group(odp_queue_t handle) { - queue_entry_t *queue; - - queue = queue_to_qentry(handle); - - return queue->s.param.sched.group; + return qentry_from_int(queue_from_ext(handle))->s.param.sched.group; } -int odp_queue_lock_count(odp_queue_t handle) +static int queue_lock_count(odp_queue_t handle) { - queue_entry_t *queue = queue_to_qentry(handle); + queue_entry_t *queue = qentry_from_int(queue_from_ext(handle)); return queue->s.param.sched.sync == ODP_SCHED_SYNC_ORDERED ? (int)queue->s.param.sched.lock_count : -1; } -odp_queue_t odp_queue_create(const char *name, const odp_queue_param_t *param) +static odp_queue_t queue_create(const char *name, + const odp_queue_param_t *param) { uint32_t i; queue_entry_t *queue; @@ -291,10 +296,10 @@ void sched_cb_queue_destroy_finalize(uint32_t queue_index) UNLOCK(&queue->s.lock); } -int odp_queue_destroy(odp_queue_t handle) +static int queue_destroy(odp_queue_t handle) { queue_entry_t *queue; - queue = queue_to_qentry(handle); + queue = qentry_from_int(queue_from_ext(handle)); if (handle == ODP_QUEUE_INVALID) return -1; @@ -343,25 +348,21 @@ int odp_queue_destroy(odp_queue_t handle) return 0; } -int odp_queue_context_set(odp_queue_t handle, void *context, - uint32_t len ODP_UNUSED) +static int queue_context_set(odp_queue_t handle, void *context, + uint32_t len ODP_UNUSED) { - queue_entry_t *queue; - queue = queue_to_qentry(handle); odp_mb_full(); - queue->s.param.context = context; + qentry_from_int(queue_from_ext(handle))->s.param.context = context; odp_mb_full(); return 0; } -void *odp_queue_context(odp_queue_t handle) +static void *queue_context(odp_queue_t handle) { - queue_entry_t *queue; - queue = queue_to_qentry(handle); - return queue->s.param.context; + return qentry_from_int(queue_from_ext(handle))->s.param.context; } -odp_queue_t odp_queue_lookup(const char *name) +static odp_queue_t queue_lookup(const char *name) { uint32_t i; @@ -384,15 +385,16 @@ odp_queue_t odp_queue_lookup(const char *name) return ODP_QUEUE_INVALID; } -static inline int enq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], +static inline int enq_multi(queue_t handle, odp_buffer_hdr_t *buf_hdr[], int num) { int sched = 0; int i, ret; + queue_entry_t *queue; odp_buffer_hdr_t *hdr, *tail, *next_hdr; - if (sched_fn->ord_enq_multi(queue->s.index, (void **)buf_hdr, num, - &ret)) + queue = qentry_from_int(handle); + if (sched_fn->ord_enq_multi(handle, (void **)buf_hdr, num, &ret)) return ret; /* Optimize the common case of single enqueue */ @@ -460,16 +462,17 @@ static inline int enq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], return num; /* All events enqueued */ } -int queue_enq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], int num) +static int _queue_enq_multi(queue_t handle, odp_buffer_hdr_t *buf_hdr[], + int num) { - return enq_multi(queue, buf_hdr, num); + return enq_multi(handle, buf_hdr, num); } -int queue_enq(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr) +static int _queue_enq(queue_t handle, odp_buffer_hdr_t *buf_hdr) { int ret; - ret = enq_multi(queue, &buf_hdr, 1); + ret = enq_multi(handle, &buf_hdr, 1); if (ret == 1) return 0; @@ -477,7 +480,7 @@ int queue_enq(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr) return -1; } -int odp_queue_enq_multi(odp_queue_t handle, const odp_event_t ev[], int num) +static int queue_enq_multi(odp_queue_t handle, const odp_event_t ev[], int num) { odp_buffer_hdr_t *buf_hdr[QUEUE_MULTI_MAX]; queue_entry_t *queue; @@ -486,33 +489,35 @@ int odp_queue_enq_multi(odp_queue_t handle, const odp_event_t ev[], int num) if (num > QUEUE_MULTI_MAX) num = QUEUE_MULTI_MAX; - queue = queue_to_qentry(handle); + queue = qentry_from_int(queue_from_ext(handle)); for (i = 0; i < num; i++) buf_hdr[i] = buf_hdl_to_hdr(odp_buffer_from_event(ev[i])); - return num == 0 ? 0 : queue->s.enqueue_multi(queue, buf_hdr, - num); + return num == 0 ? 0 : queue->s.enqueue_multi(qentry_to_int(queue), + buf_hdr, num); } -int odp_queue_enq(odp_queue_t handle, odp_event_t ev) +static int queue_enq(odp_queue_t handle, odp_event_t ev) { odp_buffer_hdr_t *buf_hdr; queue_entry_t *queue; - queue = queue_to_qentry(handle); + queue = qentry_from_int(queue_from_ext(handle)); buf_hdr = buf_hdl_to_hdr(odp_buffer_from_event(ev)); - return queue->s.enqueue(queue, buf_hdr); + return queue->s.enqueue(qentry_to_int(queue), buf_hdr); } -static inline int deq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], +static inline int deq_multi(queue_t handle, odp_buffer_hdr_t *buf_hdr[], int num) { odp_buffer_hdr_t *hdr, *next; int i, j; + queue_entry_t *queue; int updated = 0; + queue = qentry_from_int(handle); LOCK(&queue->s.lock); if (odp_unlikely(queue->s.status < QUEUE_STATUS_READY)) { /* Bad queue, or queue has been destroyed. @@ -578,17 +583,18 @@ static inline int deq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], return i; } -int queue_deq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], int num) +static int _queue_deq_multi(queue_t handle, odp_buffer_hdr_t *buf_hdr[], + int num) { - return deq_multi(queue, buf_hdr, num); + return deq_multi(handle, buf_hdr, num); } -odp_buffer_hdr_t *queue_deq(queue_entry_t *queue) +static odp_buffer_hdr_t *_queue_deq(queue_t handle) { odp_buffer_hdr_t *buf_hdr = NULL; int ret; - ret = deq_multi(queue, &buf_hdr, 1); + ret = deq_multi(handle, &buf_hdr, 1); if (ret == 1) return buf_hdr; @@ -596,7 +602,7 @@ odp_buffer_hdr_t *queue_deq(queue_entry_t *queue) return NULL; } -int odp_queue_deq_multi(odp_queue_t handle, odp_event_t events[], int num) +static int queue_deq_multi(odp_queue_t handle, odp_event_t events[], int num) { queue_entry_t *queue; odp_buffer_hdr_t *buf_hdr[QUEUE_MULTI_MAX]; @@ -605,9 +611,9 @@ int odp_queue_deq_multi(odp_queue_t handle, odp_event_t events[], int num) if (num > QUEUE_MULTI_MAX) num = QUEUE_MULTI_MAX; - queue = queue_to_qentry(handle); + queue = qentry_from_int(queue_from_ext(handle)); - ret = queue->s.dequeue_multi(queue, buf_hdr, num); + ret = queue->s.dequeue_multi(qentry_to_int(queue), buf_hdr, num); for (i = 0; i < ret; i++) events[i] = odp_buffer_to_event(buf_hdr[i]->handle.handle); @@ -616,13 +622,13 @@ int odp_queue_deq_multi(odp_queue_t handle, odp_event_t events[], int num) } -odp_event_t odp_queue_deq(odp_queue_t handle) +static odp_event_t queue_deq(odp_queue_t handle) { queue_entry_t *queue; odp_buffer_hdr_t *buf_hdr; - queue = queue_to_qentry(handle); - buf_hdr = queue->s.dequeue(queue); + queue = qentry_from_int(queue_from_ext(handle)); + buf_hdr = queue->s.dequeue(qentry_to_int(queue)); if (buf_hdr) return odp_buffer_to_event(buf_hdr->handle.handle); @@ -640,7 +646,7 @@ void queue_unlock(queue_entry_t *queue) UNLOCK(&queue->s.lock); } -void odp_queue_param_init(odp_queue_param_t *params) +static void queue_param_init(odp_queue_param_t *params) { memset(params, 0, sizeof(odp_queue_param_t)); params->type = ODP_QUEUE_TYPE_PLAIN; @@ -651,7 +657,7 @@ void odp_queue_param_init(odp_queue_param_t *params) params->sched.group = ODP_SCHED_GROUP_ALL; } -int odp_queue_info(odp_queue_t handle, odp_queue_info_t *info) +static int queue_info(odp_queue_t handle, odp_queue_info_t *info) { uint32_t queue_id; queue_entry_t *queue; @@ -730,7 +736,7 @@ int sched_cb_queue_deq_multi(uint32_t queue_index, odp_event_t ev[], int num) queue_entry_t *qe = get_qentry(queue_index); odp_buffer_hdr_t *buf_hdr[num]; - ret = deq_multi(qe, buf_hdr, num); + ret = deq_multi(qentry_to_int(qe), buf_hdr, num); if (ret > 0) for (i = 0; i < ret; i++) @@ -765,7 +771,112 @@ int sched_cb_queue_empty(uint32_t queue_index) return ret; } -uint64_t odp_queue_to_u64(odp_queue_t hdl) +static uint64_t queue_to_u64(odp_queue_t hdl) { return _odp_pri(hdl); } + +static odp_pktout_queue_t queue_get_pktout(queue_t handle) +{ + return qentry_from_int(handle)->s.pktout; +} + +static void queue_set_pktout(queue_t handle, odp_pktio_t pktio, int index) +{ + qentry_from_int(handle)->s.pktout.pktio = pktio; + qentry_from_int(handle)->s.pktout.index = index; +} + +static odp_pktin_queue_t queue_get_pktin(queue_t handle) +{ + return qentry_from_int(handle)->s.pktin; +} + +static void queue_set_pktin(queue_t handle, odp_pktio_t pktio, int index) +{ + qentry_from_int(handle)->s.pktin.pktio = pktio; + qentry_from_int(handle)->s.pktin.index = index; +} + +static void queue_set_enq_func(queue_t handle, queue_enq_fn_t func) +{ + qentry_from_int(handle)->s.enqueue = func; +} + +static void queue_set_enq_multi_func(queue_t handle, queue_enq_multi_fn_t func) +{ + qentry_from_int(handle)->s.enqueue_multi = func; +} + +static void queue_set_deq_func(queue_t handle, queue_deq_fn_t func) +{ + qentry_from_int(handle)->s.dequeue = func; +} + +static void queue_set_deq_multi_func(queue_t handle, queue_deq_multi_fn_t func) +{ + qentry_from_int(handle)->s.dequeue_multi = func; +} + +static void queue_set_type(queue_t handle, odp_queue_type_t type) +{ + qentry_from_int(handle)->s.type = type; +} + +static queue_t queue_from_ext(odp_queue_t handle) +{ + uint32_t queue_id; + + queue_id = queue_to_id(handle); + return qentry_to_int(get_qentry(queue_id)); +} + +static odp_queue_t queue_to_ext(queue_t handle) +{ + return qentry_from_int(handle)->s.handle; +} + +/* API functions */ +queue_api_t queue_default_api = { + .queue_create = queue_create, + .queue_destroy = queue_destroy, + .queue_lookup = queue_lookup, + .queue_capability = queue_capability, + .queue_context_set = queue_context_set, + .queue_context = queue_context, + .queue_enq = queue_enq, + .queue_enq_multi = queue_enq_multi, + .queue_deq = queue_deq, + .queue_deq_multi = queue_deq_multi, + .queue_type = queue_type, + .queue_sched_type = queue_sched_type, + .queue_sched_prio = queue_sched_prio, + .queue_sched_group = queue_sched_group, + .queue_lock_count = queue_lock_count, + .queue_to_u64 = queue_to_u64, + .queue_param_init = queue_param_init, + .queue_info = queue_info +}; + +/* Functions towards internal components */ +queue_fn_t queue_default_fn = { + .init_global = queue_init_global, + .term_global = queue_term_global, + .init_local = queue_init_local, + .term_local = queue_term_local, + .from_ext = queue_from_ext, + .to_ext = queue_to_ext, + .enq = _queue_enq, + .enq_multi = _queue_enq_multi, + .deq = _queue_deq, + .deq_multi = _queue_deq_multi, + .get_pktout = queue_get_pktout, + .set_pktout = queue_set_pktout, + .get_pktin = queue_get_pktin, + .set_pktin = queue_set_pktin, + .set_enq_fn = queue_set_enq_func, + .set_enq_multi_fn = queue_set_enq_multi_func, + .set_deq_fn = queue_set_deq_func, + .set_deq_multi_fn = queue_set_deq_multi_func, + .set_type = queue_set_type +}; diff --git a/platform/linux-generic/odp_queue_if.c b/platform/linux-generic/odp_queue_if.c new file mode 100644 index 00000000..c91f00eb --- /dev/null +++ b/platform/linux-generic/odp_queue_if.c @@ -0,0 +1,103 @@ +/* Copyright (c) 2017, ARM Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include <odp_queue_if.h> + +extern const queue_api_t queue_default_api; +extern const queue_fn_t queue_default_fn; + +const queue_api_t *queue_api = &queue_default_api; +const queue_fn_t *queue_fn = &queue_default_fn; + +odp_queue_t odp_queue_create(const char *name, const odp_queue_param_t *param) +{ + return queue_api->queue_create(name, param); +} + +int odp_queue_destroy(odp_queue_t queue) +{ + return queue_api->queue_destroy(queue); +} + +odp_queue_t odp_queue_lookup(const char *name) +{ + return queue_api->queue_lookup(name); +} + +int odp_queue_capability(odp_queue_capability_t *capa) +{ + return queue_api->queue_capability(capa); +} + +int odp_queue_context_set(odp_queue_t queue, void *context, uint32_t len) +{ + return queue_api->queue_context_set(queue, context, len); +} + +void *odp_queue_context(odp_queue_t queue) +{ + return queue_api->queue_context(queue); +} + +int odp_queue_enq(odp_queue_t queue, odp_event_t ev) +{ + return queue_api->queue_enq(queue, ev); +} + +int odp_queue_enq_multi(odp_queue_t queue, const odp_event_t events[], int num) +{ + return queue_api->queue_enq_multi(queue, events, num); +} + +odp_event_t odp_queue_deq(odp_queue_t queue) +{ + return queue_api->queue_deq(queue); +} + +int odp_queue_deq_multi(odp_queue_t queue, odp_event_t events[], int num) +{ + return queue_api->queue_deq_multi(queue, events, num); +} + +odp_queue_type_t odp_queue_type(odp_queue_t queue) +{ + return queue_api->queue_type(queue); +} + +odp_schedule_sync_t odp_queue_sched_type(odp_queue_t queue) +{ + return queue_api->queue_sched_type(queue); +} + +odp_schedule_prio_t odp_queue_sched_prio(odp_queue_t queue) +{ + return queue_api->queue_sched_prio(queue); +} + +odp_schedule_group_t odp_queue_sched_group(odp_queue_t queue) +{ + return queue_api->queue_sched_group(queue); +} + +int odp_queue_lock_count(odp_queue_t queue) +{ + return queue_api->queue_lock_count(queue); +} + +uint64_t odp_queue_to_u64(odp_queue_t hdl) +{ + return queue_api->queue_to_u64(hdl); +} + +void odp_queue_param_init(odp_queue_param_t *param) +{ + return queue_api->queue_param_init(param); +} + +int odp_queue_info(odp_queue_t queue, odp_queue_info_t *info) +{ + return queue_api->queue_info(queue, info); +} diff --git a/platform/linux-generic/odp_schedule.c b/platform/linux-generic/odp_schedule.c index c4567d81..011d4dc4 100644 --- a/platform/linux-generic/odp_schedule.c +++ b/platform/linux-generic/odp_schedule.c @@ -646,7 +646,7 @@ static inline void ordered_stash_release(void) buf_hdr = sched_local.ordered.stash[i].buf_hdr; num = sched_local.ordered.stash[i].num; - queue_enq_multi(queue, buf_hdr, num); + queue_fn->enq_multi(qentry_to_int(queue), buf_hdr, num); } sched_local.ordered.stash_num = 0; } @@ -712,12 +712,12 @@ static inline int copy_events(odp_event_t out_ev[], unsigned int max) return i; } -static int schedule_ord_enq_multi(uint32_t queue_index, void *buf_hdr[], +static int schedule_ord_enq_multi(queue_t handle, void *buf_hdr[], int num, int *ret) { int i; uint32_t stash_num = sched_local.ordered.stash_num; - queue_entry_t *dst_queue = get_qentry(queue_index); + queue_entry_t *dst_queue = qentry_from_int(handle); queue_entry_t *src_queue = sched_local.ordered.src_queue; if (!sched_local.ordered.src_queue || sched_local.ordered.in_order) diff --git a/platform/linux-generic/odp_schedule_iquery.c b/platform/linux-generic/odp_schedule_iquery.c index 75470aff..bdf1a460 100644 --- a/platform/linux-generic/odp_schedule_iquery.c +++ b/platform/linux-generic/odp_schedule_iquery.c @@ -1108,7 +1108,7 @@ static inline void ordered_stash_release(void) buf_hdr = thread_local.ordered.stash[i].buf_hdr; num = thread_local.ordered.stash[i].num; - queue_enq_multi(queue, buf_hdr, num); + queue_fn->enq_multi(qentry_to_int(queue), buf_hdr, num); } thread_local.ordered.stash_num = 0; } @@ -1159,12 +1159,12 @@ static inline void schedule_release_context(void) schedule_release_atomic(); } -static int schedule_ord_enq_multi(uint32_t queue_index, void *buf_hdr[], +static int schedule_ord_enq_multi(queue_t handle, void *buf_hdr[], int num, int *ret) { int i; uint32_t stash_num = thread_local.ordered.stash_num; - queue_entry_t *dst_queue = get_qentry(queue_index); + queue_entry_t *dst_queue = qentry_from_int(handle); queue_entry_t *src_queue = thread_local.ordered.src_queue; if (!thread_local.ordered.src_queue || thread_local.ordered.in_order) diff --git a/platform/linux-generic/odp_schedule_sp.c b/platform/linux-generic/odp_schedule_sp.c index 0fd4d87d..91d70e3a 100644 --- a/platform/linux-generic/odp_schedule_sp.c +++ b/platform/linux-generic/odp_schedule_sp.c @@ -414,10 +414,10 @@ static int unsched_queue(uint32_t qi ODP_UNUSED) return 0; } -static int ord_enq_multi(uint32_t queue_index, void *buf_hdr[], int num, +static int ord_enq_multi(queue_t handle, void *buf_hdr[], int num, int *ret) { - (void)queue_index; + (void)handle; (void)buf_hdr; (void)num; (void)ret; diff --git a/platform/linux-generic/odp_traffic_mngr.c b/platform/linux-generic/odp_traffic_mngr.c index a93b3ba9..8b5f3187 100644 --- a/platform/linux-generic/odp_traffic_mngr.c +++ b/platform/linux-generic/odp_traffic_mngr.c @@ -102,7 +102,7 @@ static odp_bool_t tm_demote_pkt_desc(tm_system_t *tm_system, tm_shaper_obj_t *timer_shaper, pkt_desc_t *demoted_pkt_desc); -static int queue_tm_reenq(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr) +static int queue_tm_reenq(queue_t queue, odp_buffer_hdr_t *buf_hdr) { odp_tm_queue_t tm_queue = MAKE_ODP_TM_QUEUE((uint8_t *)queue - offsetof(tm_queue_obj_t, @@ -112,7 +112,7 @@ static int queue_tm_reenq(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr) return odp_tm_enq(tm_queue, pkt); } -static int queue_tm_reenq_multi(queue_entry_t *queue ODP_UNUSED, +static int queue_tm_reenq_multi(queue_t queue ODP_UNUSED, odp_buffer_hdr_t *buf[] ODP_UNUSED, int num ODP_UNUSED) { @@ -3882,6 +3882,7 @@ odp_tm_queue_t odp_tm_queue_create(odp_tm_t odp_tm, tm_queue_obj_t *tm_queue_obj; tm_wred_node_t *tm_wred_node; odp_tm_queue_t odp_tm_queue; + odp_queue_t queue; odp_tm_wred_t wred_profile; tm_system_t *tm_system; uint32_t color; @@ -3918,9 +3919,17 @@ odp_tm_queue_t odp_tm_queue_create(odp_tm_t odp_tm, tm_queue_obj->pkt = ODP_PACKET_INVALID; odp_ticketlock_init(&tm_wred_node->tm_wred_node_lock); - tm_queue_obj->tm_qentry.s.type = QUEUE_TYPE_TM; - tm_queue_obj->tm_qentry.s.enqueue = queue_tm_reenq; - tm_queue_obj->tm_qentry.s.enqueue_multi = queue_tm_reenq_multi; + queue = odp_queue_create(NULL, NULL); + if (queue == ODP_QUEUE_INVALID) { + free(tm_wred_node); + free(tm_queue_obj); + return ODP_TM_INVALID; + } + tm_queue_obj->tm_qentry = queue_fn->from_ext(queue); + queue_fn->set_type(tm_queue_obj->tm_qentry, QUEUE_TYPE_TM); + queue_fn->set_enq_fn(tm_queue_obj->tm_qentry, queue_tm_reenq); + queue_fn->set_enq_multi_fn(tm_queue_obj->tm_qentry, + queue_tm_reenq_multi); tm_system->queue_num_tbl[tm_queue_obj->queue_num - 1] = tm_queue_obj; odp_ticketlock_lock(&tm_system->tm_system_lock); @@ -3992,6 +4001,8 @@ int odp_tm_queue_destroy(odp_tm_queue_t tm_queue) odp_ticketlock_lock(&tm_system->tm_system_lock); tm_system->queue_num_tbl[tm_queue_obj->queue_num - 1] = NULL; + odp_queue_destroy(queue_fn->to_ext(tm_queue_obj->tm_qentry)); + /* First delete any associated tm_wred_node and then the tm_queue_obj * itself */ free(tm_queue_obj->tm_wred_node); diff --git a/platform/linux-generic/pktio/loop.c b/platform/linux-generic/pktio/loop.c index cc636eba..c825393a 100644 --- a/platform/linux-generic/pktio/loop.c +++ b/platform/linux-generic/pktio/loop.c @@ -11,6 +11,7 @@ #include <odp_classification_internal.h> #include <odp_debug_internal.h> #include <odp/api/hints.h> +#include <odp_queue_if.h> #include <protocols/eth.h> #include <protocols/ip.h> @@ -55,7 +56,7 @@ static int loopback_recv(pktio_entry_t *pktio_entry, int index ODP_UNUSED, { int nbr, i; odp_buffer_hdr_t *hdr_tbl[QUEUE_MULTI_MAX]; - queue_entry_t *qentry; + queue_t queue; odp_packet_hdr_t *pkt_hdr; odp_packet_hdr_t parsed_hdr; odp_packet_t pkt; @@ -69,8 +70,8 @@ static int loopback_recv(pktio_entry_t *pktio_entry, int index ODP_UNUSED, odp_ticketlock_lock(&pktio_entry->s.rxl); - qentry = queue_to_qentry(pktio_entry->s.pkt_loop.loopq); - nbr = queue_deq_multi(qentry, hdr_tbl, len); + queue = queue_fn->from_ext(pktio_entry->s.pkt_loop.loopq); + nbr = queue_fn->deq_multi(queue, hdr_tbl, len); if (pktio_entry->s.config.pktin.bit.ts_all || pktio_entry->s.config.pktin.bit.ts_ptp) { @@ -154,7 +155,7 @@ static int loopback_send(pktio_entry_t *pktio_entry, int index ODP_UNUSED, const odp_packet_t pkt_tbl[], int len) { odp_buffer_hdr_t *hdr_tbl[QUEUE_MULTI_MAX]; - queue_entry_t *qentry; + queue_t queue; int i; int ret; uint32_t bytes = 0; @@ -169,8 +170,8 @@ static int loopback_send(pktio_entry_t *pktio_entry, int index ODP_UNUSED, odp_ticketlock_lock(&pktio_entry->s.txl); - qentry = queue_to_qentry(pktio_entry->s.pkt_loop.loopq); - ret = queue_enq_multi(qentry, hdr_tbl, len); + queue = queue_fn->from_ext(pktio_entry->s.pkt_loop.loopq); + ret = queue_fn->enq_multi(queue, hdr_tbl, len); if (ret > 0) { pktio_entry->s.stats.out_ucast_pkts += ret; ----------------------------------------------------------------------- Summary of changes: platform/linux-generic/Makefile.am | 3 +- .../include/odp_classification_datamodel.h | 4 +- .../linux-generic/include/odp_config_internal.h | 2 +- .../linux-generic/include/odp_packet_internal.h | 3 +- .../linux-generic/include/odp_packet_io_internal.h | 2 + .../linux-generic/include/odp_packet_io_queue.h | 50 --- platform/linux-generic/include/odp_queue_if.h | 100 ++++++ .../linux-generic/include/odp_queue_internal.h | 48 +-- platform/linux-generic/include/odp_schedule_if.h | 15 +- .../include/odp_traffic_mngr_internal.h | 4 +- platform/linux-generic/odp_classification.c | 18 +- platform/linux-generic/odp_init.c | 17 +- platform/linux-generic/odp_packet_io.c | 145 ++++---- platform/linux-generic/odp_queue.c | 376 ++++++++++++--------- platform/linux-generic/odp_queue_if.c | 103 ++++++ platform/linux-generic/odp_schedule.c | 164 +++++---- platform/linux-generic/odp_schedule_iquery.c | 162 +++++---- platform/linux-generic/odp_schedule_sp.c | 22 +- platform/linux-generic/odp_traffic_mngr.c | 23 +- platform/linux-generic/pktio/loop.c | 13 +- 20 files changed, 766 insertions(+), 508 deletions(-) delete mode 100644 platform/linux-generic/include/odp_packet_io_queue.h create mode 100644 platform/linux-generic/include/odp_queue_if.h create mode 100644 platform/linux-generic/odp_queue_if.c hooks/post-receive --

9 years, 1 month

[COMMIT] branch api-next updated. v1.15.0.0-411-g0275cc02

by git＠git-us.linaro.org

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, api-next has been updated via 0275cc02d5cdb54b955cfce1988f5ee158ad0423 (commit) via d30dc0774647b3f635afea67e56147b8e6e3f24e (commit) via bf5d48ee13aa4fb737ee430038d5f2b865f45d47 (commit) via 1ba2bcd7b9ebfa1a5d8dda257c7b309d3553fa03 (commit) via 8869e18f490d0ffc35e0359a92ef96eacbf5b0b7 (commit) via 91c4f94f1ed01ed22b7cd489016016b0599f4065 (commit) via 1cb0a8863eb8309c3e532a168fed34d6e3257218 (commit) via faf97a341df8e92fd692029fe2c6c353f043950b (commit) from c7718962c6633c80eb71a0400d89c31f11f88045 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 0275cc02d5cdb54b955cfce1988f5ee158ad0423 Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Tue Jul 4 12:53:43 2017 +0300 linux-gen: queue: remove type set from interface TM should not modify queue type. Remove type_set queue interface function. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_queue_if.h b/platform/linux-generic/include/odp_queue_if.h index 7ba5c3c5..410c6b79 100644 --- a/platform/linux-generic/include/odp_queue_if.h +++ b/platform/linux-generic/include/odp_queue_if.h @@ -71,7 +71,6 @@ typedef void (*queue_set_enq_deq_fn_t)(queue_t q_int, queue_enq_multi_fn_t enq_multi, queue_deq_fn_t deq, queue_deq_multi_fn_t deq_multi); -typedef void (*queue_set_type_fn_t)(queue_t q_int, odp_queue_type_t type); /* Queue functions towards other internal components */ typedef struct { @@ -90,7 +89,6 @@ typedef struct { queue_get_pktin_fn_t get_pktin; queue_set_pktin_fn_t set_pktin; queue_set_enq_deq_fn_t set_enq_deq_fn; - queue_set_type_fn_t set_type; } queue_fn_t; extern const queue_fn_t *queue_fn; diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c index ec225d00..d52814b2 100644 --- a/platform/linux-generic/odp_queue.c +++ b/platform/linux-generic/odp_queue.c @@ -769,11 +769,6 @@ static void queue_set_enq_deq_func(queue_t q_int, qentry->s.dequeue_multi = deq_multi; } -static void queue_set_type(queue_t q_int, odp_queue_type_t type) -{ - qentry_from_int(q_int)->s.type = type; -} - static queue_t queue_from_ext(odp_queue_t handle) { return qentry_to_int(handle_to_qentry(handle)); @@ -822,6 +817,5 @@ queue_fn_t queue_default_fn = { .set_pktout = queue_set_pktout, .get_pktin = queue_get_pktin, .set_pktin = queue_set_pktin, - .set_enq_deq_fn = queue_set_enq_deq_func, - .set_type = queue_set_type + .set_enq_deq_fn = queue_set_enq_deq_func }; diff --git a/platform/linux-generic/odp_queue_scalable.c b/platform/linux-generic/odp_queue_scalable.c index 14fa1833..52f3915f 100644 --- a/platform/linux-generic/odp_queue_scalable.c +++ b/platform/linux-generic/odp_queue_scalable.c @@ -956,11 +956,6 @@ static void queue_set_enq_deq_func(queue_t handle, qentry_from_int(handle)->s.dequeue_multi = deq_multi; } -static void queue_set_type(queue_t handle, odp_queue_type_t type) -{ - qentry_from_int(handle)->s.type = type; -} - static queue_t queue_from_ext(odp_queue_t handle) { uint32_t queue_id; @@ -1012,6 +1007,5 @@ queue_fn_t queue_scalable_fn = { .set_pktout = queue_set_pktout, .get_pktin = queue_get_pktin, .set_pktin = queue_set_pktin, - .set_enq_deq_fn = queue_set_enq_deq_func, - .set_type = queue_set_type + .set_enq_deq_fn = queue_set_enq_deq_func }; diff --git a/platform/linux-generic/odp_traffic_mngr.c b/platform/linux-generic/odp_traffic_mngr.c index 1f254d75..ec2d331f 100644 --- a/platform/linux-generic/odp_traffic_mngr.c +++ b/platform/linux-generic/odp_traffic_mngr.c @@ -37,9 +37,6 @@ static const pkt_desc_t EMPTY_PKT_DESC = { .word = 0 }; #define MAX_PRIORITIES ODP_TM_MAX_PRIORITIES #define NUM_SHAPER_COLORS ODP_NUM_SHAPER_COLORS -/* Traffic manager queue */ -#define QUEUE_TYPE_TM 4 - static tm_prop_t basic_prop_tbl[MAX_PRIORITIES][NUM_SHAPER_COLORS] = { [0] = { [ODP_TM_SHAPER_GREEN] = { 0, DECR_BOTH }, @@ -3926,7 +3923,6 @@ odp_tm_queue_t odp_tm_queue_create(odp_tm_t odp_tm, return ODP_TM_INVALID; } tm_queue_obj->tm_qentry = queue_fn->from_ext(queue); - queue_fn->set_type(tm_queue_obj->tm_qentry, QUEUE_TYPE_TM); queue_fn->set_enq_deq_fn(tm_queue_obj->tm_qentry, queue_tm_reenq, queue_tm_reenq_multi, NULL, NULL); commit d30dc0774647b3f635afea67e56147b8e6e3f24e Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Tue Jul 4 12:53:42 2017 +0300 linux-gen: queue: combine queue interface set functions Decrease queue interface size by combining four queue operation set functions into a single function. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_queue_if.h b/platform/linux-generic/include/odp_queue_if.h index b5cbd515..7ba5c3c5 100644 --- a/platform/linux-generic/include/odp_queue_if.h +++ b/platform/linux-generic/include/odp_queue_if.h @@ -66,12 +66,11 @@ typedef void (*queue_set_pktout_fn_t)(queue_t q_int, odp_pktio_t pktio, typedef odp_pktin_queue_t (*queue_get_pktin_fn_t)(queue_t q_int); typedef void (*queue_set_pktin_fn_t)(queue_t q_int, odp_pktio_t pktio, int index); -typedef void (*queue_set_enq_fn_t)(queue_t q_int, queue_enq_fn_t func); -typedef void (*queue_set_enq_multi_fn_t)(queue_t q_int, - queue_enq_multi_fn_t func); -typedef void (*queue_set_deq_fn_t)(queue_t q_int, queue_deq_fn_t func); -typedef void (*queue_set_deq_multi_fn_t)(queue_t q_int, - queue_deq_multi_fn_t func); +typedef void (*queue_set_enq_deq_fn_t)(queue_t q_int, + queue_enq_fn_t enq, + queue_enq_multi_fn_t enq_multi, + queue_deq_fn_t deq, + queue_deq_multi_fn_t deq_multi); typedef void (*queue_set_type_fn_t)(queue_t q_int, odp_queue_type_t type); /* Queue functions towards other internal components */ @@ -90,10 +89,7 @@ typedef struct { queue_set_pktout_fn_t set_pktout; queue_get_pktin_fn_t get_pktin; queue_set_pktin_fn_t set_pktin; - queue_set_enq_fn_t set_enq_fn; - queue_set_enq_multi_fn_t set_enq_multi_fn; - queue_set_deq_fn_t set_deq_fn; - queue_set_deq_multi_fn_t set_deq_multi_fn; + queue_set_enq_deq_fn_t set_enq_deq_fn; queue_set_type_fn_t set_type; } queue_fn_t; diff --git a/platform/linux-generic/odp_packet_io.c b/platform/linux-generic/odp_packet_io.c index 0d825f91..6a181f53 100644 --- a/platform/linux-generic/odp_packet_io.c +++ b/platform/linux-generic/odp_packet_io.c @@ -1154,30 +1154,30 @@ int odp_pktio_stats_reset(odp_pktio_t pktio) return ret; } -static int pktin_enqueue(queue_t q_int ODP_UNUSED, - odp_buffer_hdr_t *buf_hdr ODP_UNUSED) +static int abort_pktin_enqueue(queue_t q_int ODP_UNUSED, + odp_buffer_hdr_t *buf_hdr ODP_UNUSED) { ODP_ABORT("attempted enqueue to a pktin queue"); return -1; } -static int pktin_enq_multi(queue_t q_int ODP_UNUSED, - odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, - int num ODP_UNUSED) +static int abort_pktin_enq_multi(queue_t q_int ODP_UNUSED, + odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, + int num ODP_UNUSED) { ODP_ABORT("attempted enqueue to a pktin queue"); return 0; } -static odp_buffer_hdr_t *pktout_dequeue(queue_t q_int ODP_UNUSED) +static odp_buffer_hdr_t *abort_pktout_dequeue(queue_t q_int ODP_UNUSED) { ODP_ABORT("attempted dequeue from a pktout queue"); return NULL; } -static int pktout_deq_multi(queue_t q_int ODP_UNUSED, - odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, - int num ODP_UNUSED) +static int abort_pktout_deq_multi(queue_t q_int ODP_UNUSED, + odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, + int num ODP_UNUSED) { ODP_ABORT("attempted dequeue from a pktout queue"); return 0; @@ -1272,12 +1272,11 @@ int odp_pktin_queue_config(odp_pktio_t pktio, if (mode == ODP_PKTIN_MODE_QUEUE) { queue_fn->set_pktin(q_int, pktio, i); - queue_fn->set_enq_fn(q_int, pktin_enqueue); - queue_fn->set_deq_fn(q_int, pktin_dequeue); - queue_fn->set_enq_multi_fn(q_int, - pktin_enq_multi); - queue_fn->set_deq_multi_fn(q_int, - pktin_deq_multi); + queue_fn->set_enq_deq_fn(q_int, + abort_pktin_enqueue, + abort_pktin_enq_multi, + pktin_dequeue, + pktin_deq_multi); } entry->s.in_queue[i].queue = queue; @@ -1401,10 +1400,11 @@ int odp_pktout_queue_config(odp_pktio_t pktio, queue_fn->set_pktout(q_int, pktio, i); /* Override default enqueue / dequeue functions */ - queue_fn->set_enq_fn(q_int, pktout_enqueue); - queue_fn->set_deq_fn(q_int, pktout_dequeue); - queue_fn->set_enq_multi_fn(q_int, pktout_enq_multi); - queue_fn->set_deq_multi_fn(q_int, pktout_deq_multi); + queue_fn->set_enq_deq_fn(q_int, + pktout_enqueue, + pktout_enq_multi, + abort_pktout_dequeue, + abort_pktout_deq_multi); entry->s.out_queue[i].queue = queue; } diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c index 60f88f09..ec225d00 100644 --- a/platform/linux-generic/odp_queue.c +++ b/platform/linux-generic/odp_queue.c @@ -748,24 +748,25 @@ static void queue_set_pktin(queue_t q_int, odp_pktio_t pktio, int index) qentry->s.pktin.index = index; } -static void queue_set_enq_func(queue_t q_int, queue_enq_fn_t func) +static void queue_set_enq_deq_func(queue_t q_int, + queue_enq_fn_t enq, + queue_enq_multi_fn_t enq_multi, + queue_deq_fn_t deq, + queue_deq_multi_fn_t deq_multi) { - qentry_from_int(q_int)->s.enqueue = func; -} + queue_entry_t *qentry = qentry_from_int(q_int); -static void queue_set_enq_multi_func(queue_t q_int, queue_enq_multi_fn_t func) -{ - qentry_from_int(q_int)->s.enqueue_multi = func; -} + if (enq) + qentry->s.enqueue = enq; -static void queue_set_deq_func(queue_t q_int, queue_deq_fn_t func) -{ - qentry_from_int(q_int)->s.dequeue = func; -} + if (enq_multi) + qentry->s.enqueue_multi = enq_multi; -static void queue_set_deq_multi_func(queue_t q_int, queue_deq_multi_fn_t func) -{ - qentry_from_int(q_int)->s.dequeue_multi = func; + if (deq) + qentry->s.dequeue = deq; + + if (deq_multi) + qentry->s.dequeue_multi = deq_multi; } static void queue_set_type(queue_t q_int, odp_queue_type_t type) @@ -821,9 +822,6 @@ queue_fn_t queue_default_fn = { .set_pktout = queue_set_pktout, .get_pktin = queue_get_pktin, .set_pktin = queue_set_pktin, - .set_enq_fn = queue_set_enq_func, - .set_enq_multi_fn = queue_set_enq_multi_func, - .set_deq_fn = queue_set_deq_func, - .set_deq_multi_fn = queue_set_deq_multi_func, + .set_enq_deq_fn = queue_set_enq_deq_func, .set_type = queue_set_type }; diff --git a/platform/linux-generic/odp_queue_scalable.c b/platform/linux-generic/odp_queue_scalable.c index 7441a2a9..14fa1833 100644 --- a/platform/linux-generic/odp_queue_scalable.c +++ b/platform/linux-generic/odp_queue_scalable.c @@ -937,24 +937,23 @@ static void queue_set_pktin(queue_t handle, odp_pktio_t pktio, int index) qentry_from_int(handle)->s.pktin.index = index; } -static void queue_set_enq_func(queue_t handle, queue_enq_fn_t func) +static void queue_set_enq_deq_func(queue_t handle, + queue_enq_fn_t enq, + queue_enq_multi_fn_t enq_multi, + queue_deq_fn_t deq, + queue_deq_multi_fn_t deq_multi) { - qentry_from_int(handle)->s.enqueue = func; -} + if (enq) + qentry_from_int(handle)->s.enqueue = enq; -static void queue_set_enq_multi_func(queue_t handle, queue_enq_multi_fn_t func) -{ - qentry_from_int(handle)->s.enqueue_multi = func; -} + if (enq_multi) + qentry_from_int(handle)->s.enqueue_multi = enq_multi; -static void queue_set_deq_func(queue_t handle, queue_deq_fn_t func) -{ - qentry_from_int(handle)->s.dequeue = func; -} + if (deq) + qentry_from_int(handle)->s.dequeue = deq; -static void queue_set_deq_multi_func(queue_t handle, queue_deq_multi_fn_t func) -{ - qentry_from_int(handle)->s.dequeue_multi = func; + if (deq_multi) + qentry_from_int(handle)->s.dequeue_multi = deq_multi; } static void queue_set_type(queue_t handle, odp_queue_type_t type) @@ -1013,9 +1012,6 @@ queue_fn_t queue_scalable_fn = { .set_pktout = queue_set_pktout, .get_pktin = queue_get_pktin, .set_pktin = queue_set_pktin, - .set_enq_fn = queue_set_enq_func, - .set_enq_multi_fn = queue_set_enq_multi_func, - .set_deq_fn = queue_set_deq_func, - .set_deq_multi_fn = queue_set_deq_multi_func, + .set_enq_deq_fn = queue_set_enq_deq_func, .set_type = queue_set_type }; diff --git a/platform/linux-generic/odp_traffic_mngr.c b/platform/linux-generic/odp_traffic_mngr.c index 8b5f3187..1f254d75 100644 --- a/platform/linux-generic/odp_traffic_mngr.c +++ b/platform/linux-generic/odp_traffic_mngr.c @@ -3927,9 +3927,9 @@ odp_tm_queue_t odp_tm_queue_create(odp_tm_t odp_tm, } tm_queue_obj->tm_qentry = queue_fn->from_ext(queue); queue_fn->set_type(tm_queue_obj->tm_qentry, QUEUE_TYPE_TM); - queue_fn->set_enq_fn(tm_queue_obj->tm_qentry, queue_tm_reenq); - queue_fn->set_enq_multi_fn(tm_queue_obj->tm_qentry, - queue_tm_reenq_multi); + queue_fn->set_enq_deq_fn(tm_queue_obj->tm_qentry, + queue_tm_reenq, queue_tm_reenq_multi, + NULL, NULL); tm_system->queue_num_tbl[tm_queue_obj->queue_num - 1] = tm_queue_obj; odp_ticketlock_lock(&tm_system->tm_system_lock); commit bf5d48ee13aa4fb737ee430038d5f2b865f45d47 Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Tue Jul 4 12:53:41 2017 +0300 linux-gen: pktio: remove unused header file odp_packet_io_queue.h is not needed anymore as the queue interface has replaced it. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index 82ab4642..b46c6501 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -175,7 +175,6 @@ noinst_HEADERS = \ ${srcdir}/include/odp_packet_internal.h \ ${srcdir}/include/odp_packet_io_internal.h \ ${srcdir}/include/odp_packet_io_ipc_internal.h \ - ${srcdir}/include/odp_packet_io_queue.h \ ${srcdir}/include/odp_packet_io_ring_internal.h \ ${srcdir}/include/odp_packet_netmap.h \ ${srcdir}/include/odp_packet_dpdk.h \ diff --git a/platform/linux-generic/include/odp_packet_io_queue.h b/platform/linux-generic/include/odp_packet_io_queue.h deleted file mode 100644 index 71333ddc..00000000 --- a/platform/linux-generic/include/odp_packet_io_queue.h +++ /dev/null @@ -1,48 +0,0 @@ -/* Copyright (c) 2013, Linaro Limited - * All rights reserved. - * - * SPDX-License-Identifier: BSD-3-Clause - */ - - -/** - * @file - * - * ODP packet IO - implementation internal - */ - -#ifndef ODP_PACKET_IO_QUEUE_H_ -#define ODP_PACKET_IO_QUEUE_H_ - -#ifdef __cplusplus -extern "C" { -#endif - -#include <odp_queue_if.h> -#include <odp_buffer_internal.h> -#include <odp_config_internal.h> - -/** Max nbr of pkts to receive in one burst (keep same as QUEUE_MULTI_MAX) */ -#define ODP_PKTIN_QUEUE_MAX_BURST CONFIG_BURST_SIZE -/* pktin_deq_multi() depends on the condition: */ -ODP_STATIC_ASSERT(ODP_PKTIN_QUEUE_MAX_BURST >= QUEUE_MULTI_MAX, - "ODP_PKTIN_DEQ_MULTI_MAX_ERROR"); - -int pktin_enqueue(queue_t queue, odp_buffer_hdr_t *buf_hdr); -odp_buffer_hdr_t *pktin_dequeue(queue_t queue); - -int pktin_enq_multi(queue_t queue, odp_buffer_hdr_t *buf_hdr[], int num); -int pktin_deq_multi(queue_t queue, odp_buffer_hdr_t *buf_hdr[], int num); - - -int pktout_enqueue(queue_t queue, odp_buffer_hdr_t *buf_hdr); -odp_buffer_hdr_t *pktout_dequeue(queue_t queue); - -int pktout_enq_multi(queue_t queue, odp_buffer_hdr_t *buf_hdr[], int num); -int pktout_deq_multi(queue_t queue, odp_buffer_hdr_t *buf_hdr[], int num); - -#ifdef __cplusplus -} -#endif - -#endif diff --git a/platform/linux-generic/odp_packet_io.c b/platform/linux-generic/odp_packet_io.c index 3ed51ac9..0d825f91 100644 --- a/platform/linux-generic/odp_packet_io.c +++ b/platform/linux-generic/odp_packet_io.c @@ -7,7 +7,6 @@ #include <odp/api/packet_io.h> #include <odp_packet_io_internal.h> -#include <odp_packet_io_queue.h> #include <odp/api/packet.h> #include <odp_packet_internal.h> #include <odp_internal.h> @@ -580,7 +579,7 @@ static inline int pktin_recv_buf(odp_pktin_queue_t queue, return num_rx; } -int pktout_enqueue(queue_t q_int, odp_buffer_hdr_t *buf_hdr) +static int pktout_enqueue(queue_t q_int, odp_buffer_hdr_t *buf_hdr) { odp_packet_t pkt = packet_from_buf_hdr(buf_hdr); int len = 1; @@ -593,13 +592,7 @@ int pktout_enqueue(queue_t q_int, odp_buffer_hdr_t *buf_hdr) return (nbr == len ? 0 : -1); } -odp_buffer_hdr_t *pktout_dequeue(queue_t q_int ODP_UNUSED) -{ - ODP_ABORT("attempted dequeue from a pktout queue"); - return NULL; -} - -int pktout_enq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int num) +static int pktout_enq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int num) { odp_packet_t pkt_tbl[QUEUE_MULTI_MAX]; int nbr; @@ -615,22 +608,7 @@ int pktout_enq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int num) return nbr; } -int pktout_deq_multi(queue_t q_int ODP_UNUSED, - odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, - int num ODP_UNUSED) -{ - ODP_ABORT("attempted dequeue from a pktout queue"); - return 0; -} - -int pktin_enqueue(queue_t q_int ODP_UNUSED, - odp_buffer_hdr_t *buf_hdr ODP_UNUSED) -{ - ODP_ABORT("attempted enqueue to a pktin queue"); - return -1; -} - -odp_buffer_hdr_t *pktin_dequeue(queue_t q_int) +static odp_buffer_hdr_t *pktin_dequeue(queue_t q_int) { odp_buffer_hdr_t *buf_hdr; odp_buffer_hdr_t *hdr_tbl[QUEUE_MULTI_MAX]; @@ -652,14 +630,7 @@ odp_buffer_hdr_t *pktin_dequeue(queue_t q_int) return buf_hdr; } -int pktin_enq_multi(queue_t q_int ODP_UNUSED, - odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, int num ODP_UNUSED) -{ - ODP_ABORT("attempted enqueue to a pktin queue"); - return 0; -} - -int pktin_deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int num) +static int pktin_deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int num) { int nbr; odp_buffer_hdr_t *hdr_tbl[QUEUE_MULTI_MAX]; @@ -1183,6 +1154,35 @@ int odp_pktio_stats_reset(odp_pktio_t pktio) return ret; } +static int pktin_enqueue(queue_t q_int ODP_UNUSED, + odp_buffer_hdr_t *buf_hdr ODP_UNUSED) +{ + ODP_ABORT("attempted enqueue to a pktin queue"); + return -1; +} + +static int pktin_enq_multi(queue_t q_int ODP_UNUSED, + odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, + int num ODP_UNUSED) +{ + ODP_ABORT("attempted enqueue to a pktin queue"); + return 0; +} + +static odp_buffer_hdr_t *pktout_dequeue(queue_t q_int ODP_UNUSED) +{ + ODP_ABORT("attempted dequeue from a pktout queue"); + return NULL; +} + +static int pktout_deq_multi(queue_t q_int ODP_UNUSED, + odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, + int num ODP_UNUSED) +{ + ODP_ABORT("attempted dequeue from a pktout queue"); + return 0; +} + int odp_pktin_queue_config(odp_pktio_t pktio, const odp_pktin_queue_param_t *param) { diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c index 4c85027b..60f88f09 100644 --- a/platform/linux-generic/odp_queue.c +++ b/platform/linux-generic/odp_queue.c @@ -19,7 +19,6 @@ #include <odp_schedule_if.h> #include <odp_config_internal.h> #include <odp_packet_io_internal.h> -#include <odp_packet_io_queue.h> #include <odp_debug_internal.h> #include <odp/api/hints.h> #include <odp/api/sync.h> diff --git a/platform/linux-generic/odp_queue_scalable.c b/platform/linux-generic/odp_queue_scalable.c index f95f5f93..7441a2a9 100644 --- a/platform/linux-generic/odp_queue_scalable.c +++ b/platform/linux-generic/odp_queue_scalable.c @@ -20,7 +20,6 @@ #include <odp_buffer_inlines.h> #include <odp_packet_io_internal.h> -#include <odp_packet_io_queue.h> #include <odp_pool_internal.h> #include <odp_queue_scalable_internal.h> #include <odp_schedule_if.h> commit 1ba2bcd7b9ebfa1a5d8dda257c7b309d3553fa03 Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Tue Jul 4 12:53:40 2017 +0300 linux-gen: pktio: convert queue handle only once Avoid queue handle conversion on fast path. Convert once after queue create and save value. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_packet_io_internal.h b/platform/linux-generic/include/odp_packet_io_internal.h index 89bb6f3a..93040681 100644 --- a/platform/linux-generic/include/odp_packet_io_internal.h +++ b/platform/linux-generic/include/odp_packet_io_internal.h @@ -24,6 +24,7 @@ extern "C" { #include <odp_align_internal.h> #include <odp_debug_internal.h> #include <odp_packet_io_ring_internal.h> +#include <odp_queue_if.h> #include <odp_config_internal.h> #include <odp/api/hints.h> @@ -165,6 +166,7 @@ struct pktio_entry { struct { odp_queue_t queue; + queue_t queue_int; odp_pktin_queue_t pktin; } in_queue[PKTIO_MAX_QUEUES]; diff --git a/platform/linux-generic/odp_packet_io.c b/platform/linux-generic/odp_packet_io.c index af1307c0..3ed51ac9 100644 --- a/platform/linux-generic/odp_packet_io.c +++ b/platform/linux-generic/odp_packet_io.c @@ -128,6 +128,7 @@ static void init_in_queues(pktio_entry_t *entry) for (i = 0; i < PKTIO_MAX_QUEUES; i++) { entry->s.in_queue[i].queue = ODP_QUEUE_INVALID; + entry->s.in_queue[i].queue_int = QUEUE_NULL; entry->s.in_queue[i].pktin = PKTIN_INVALID; } } @@ -305,6 +306,7 @@ static void destroy_in_queues(pktio_entry_t *entry, int num) if (entry->s.in_queue[i].queue != ODP_QUEUE_INVALID) { odp_queue_destroy(entry->s.in_queue[i].queue); entry->s.in_queue[i].queue = ODP_QUEUE_INVALID; + entry->s.in_queue[i].queue_int = QUEUE_NULL; } } } @@ -708,8 +710,7 @@ int sched_cb_pktin_poll(int pktio_index, int num_queue, int index[]) } for (idx = 0; idx < num_queue; idx++) { - queue_t qentry; - odp_queue_t queue; + queue_t q_int; odp_pktin_queue_t pktin = entry->s.in_queue[index[idx]].pktin; num = pktin_recv_buf(pktin, hdr_tbl, QUEUE_MULTI_MAX); @@ -722,9 +723,8 @@ int sched_cb_pktin_poll(int pktio_index, int num_queue, int index[]) return -1; } - queue = entry->s.in_queue[index[idx]].queue; - qentry = queue_fn->from_ext(queue); - queue_fn->enq_multi(qentry, hdr_tbl, num); + q_int = entry->s.in_queue[index[idx]].queue_int; + queue_fn->enq_multi(q_int, hdr_tbl, num); } return 0; @@ -1281,8 +1281,11 @@ int odp_pktin_queue_config(odp_pktio_t pktio, } entry->s.in_queue[i].queue = queue; + entry->s.in_queue[i].queue_int = q_int; + } else { entry->s.in_queue[i].queue = ODP_QUEUE_INVALID; + entry->s.in_queue[i].queue_int = QUEUE_NULL; } entry->s.in_queue[i].pktin.index = i; commit 8869e18f490d0ffc35e0359a92ef96eacbf5b0b7 Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Tue Jul 4 12:53:39 2017 +0300 linux-gen: pktio: dont use qentry for queue_t These are not any more queue_entry_t* but queue_t. Align variable names with the new type. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/odp_packet_io.c b/platform/linux-generic/odp_packet_io.c index 6ff35046..af1307c0 100644 --- a/platform/linux-generic/odp_packet_io.c +++ b/platform/linux-generic/odp_packet_io.c @@ -578,42 +578,42 @@ static inline int pktin_recv_buf(odp_pktin_queue_t queue, return num_rx; } -int pktout_enqueue(queue_t qentry, odp_buffer_hdr_t *buf_hdr) +int pktout_enqueue(queue_t q_int, odp_buffer_hdr_t *buf_hdr) { odp_packet_t pkt = packet_from_buf_hdr(buf_hdr); int len = 1; int nbr; - if (sched_fn->ord_enq_multi(qentry, (void **)buf_hdr, len, &nbr)) + if (sched_fn->ord_enq_multi(q_int, (void **)buf_hdr, len, &nbr)) return (nbr == len ? 0 : -1); - nbr = odp_pktout_send(queue_fn->get_pktout(qentry), &pkt, len); + nbr = odp_pktout_send(queue_fn->get_pktout(q_int), &pkt, len); return (nbr == len ? 0 : -1); } -odp_buffer_hdr_t *pktout_dequeue(queue_t qentry ODP_UNUSED) +odp_buffer_hdr_t *pktout_dequeue(queue_t q_int ODP_UNUSED) { ODP_ABORT("attempted dequeue from a pktout queue"); return NULL; } -int pktout_enq_multi(queue_t qentry, odp_buffer_hdr_t *buf_hdr[], int num) +int pktout_enq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int num) { odp_packet_t pkt_tbl[QUEUE_MULTI_MAX]; int nbr; int i; - if (sched_fn->ord_enq_multi(qentry, (void **)buf_hdr, num, &nbr)) + if (sched_fn->ord_enq_multi(q_int, (void **)buf_hdr, num, &nbr)) return nbr; for (i = 0; i < num; ++i) pkt_tbl[i] = packet_from_buf_hdr(buf_hdr[i]); - nbr = odp_pktout_send(queue_fn->get_pktout(qentry), pkt_tbl, num); + nbr = odp_pktout_send(queue_fn->get_pktout(q_int), pkt_tbl, num); return nbr; } -int pktout_deq_multi(queue_t qentry ODP_UNUSED, +int pktout_deq_multi(queue_t q_int ODP_UNUSED, odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, int num ODP_UNUSED) { @@ -621,49 +621,49 @@ int pktout_deq_multi(queue_t qentry ODP_UNUSED, return 0; } -int pktin_enqueue(queue_t qentry ODP_UNUSED, +int pktin_enqueue(queue_t q_int ODP_UNUSED, odp_buffer_hdr_t *buf_hdr ODP_UNUSED) { ODP_ABORT("attempted enqueue to a pktin queue"); return -1; } -odp_buffer_hdr_t *pktin_dequeue(queue_t qentry) +odp_buffer_hdr_t *pktin_dequeue(queue_t q_int) { odp_buffer_hdr_t *buf_hdr; odp_buffer_hdr_t *hdr_tbl[QUEUE_MULTI_MAX]; int pkts; - buf_hdr = queue_fn->deq(qentry); + buf_hdr = queue_fn->deq(q_int); if (buf_hdr != NULL) return buf_hdr; - pkts = pktin_recv_buf(queue_fn->get_pktin(qentry), + pkts = pktin_recv_buf(queue_fn->get_pktin(q_int), hdr_tbl, QUEUE_MULTI_MAX); if (pkts <= 0) return NULL; if (pkts > 1) - queue_fn->enq_multi(qentry, &hdr_tbl[1], pkts - 1); + queue_fn->enq_multi(q_int, &hdr_tbl[1], pkts - 1); buf_hdr = hdr_tbl[0]; return buf_hdr; } -int pktin_enq_multi(queue_t qentry ODP_UNUSED, +int pktin_enq_multi(queue_t q_int ODP_UNUSED, odp_buffer_hdr_t *buf_hdr[] ODP_UNUSED, int num ODP_UNUSED) { ODP_ABORT("attempted enqueue to a pktin queue"); return 0; } -int pktin_deq_multi(queue_t qentry, odp_buffer_hdr_t *buf_hdr[], int num) +int pktin_deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int num) { int nbr; odp_buffer_hdr_t *hdr_tbl[QUEUE_MULTI_MAX]; int pkts, i, j; - nbr = queue_fn->deq_multi(qentry, buf_hdr, num); + nbr = queue_fn->deq_multi(q_int, buf_hdr, num); if (odp_unlikely(nbr > num)) ODP_ABORT("queue_deq_multi req: %d, returned %d\n", num, nbr); @@ -673,7 +673,7 @@ int pktin_deq_multi(queue_t qentry, odp_buffer_hdr_t *buf_hdr[], int num) if (nbr == num) return nbr; - pkts = pktin_recv_buf(queue_fn->get_pktin(qentry), + pkts = pktin_recv_buf(queue_fn->get_pktin(q_int), hdr_tbl, QUEUE_MULTI_MAX); if (pkts <= 0) return nbr; @@ -686,7 +686,7 @@ int pktin_deq_multi(queue_t qentry, odp_buffer_hdr_t *buf_hdr[], int num) hdr_tbl[j] = hdr_tbl[i]; if (j) - queue_fn->enq_multi(qentry, hdr_tbl, j); + queue_fn->enq_multi(q_int, hdr_tbl, j); return nbr; } @@ -1193,6 +1193,7 @@ int odp_pktin_queue_config(odp_pktio_t pktio, unsigned i; int rc; odp_queue_t queue; + queue_t q_int; odp_pktin_queue_param_t default_param; if (param == NULL) { @@ -1267,16 +1268,15 @@ int odp_pktin_queue_config(odp_pktio_t pktio, return -1; } - if (mode == ODP_PKTIN_MODE_QUEUE) { - queue_t qentry; + q_int = queue_fn->from_ext(queue); - qentry = queue_fn->from_ext(queue); - queue_fn->set_pktin(qentry, pktio, i); - queue_fn->set_enq_fn(qentry, pktin_enqueue); - queue_fn->set_deq_fn(qentry, pktin_dequeue); - queue_fn->set_enq_multi_fn(qentry, + if (mode == ODP_PKTIN_MODE_QUEUE) { + queue_fn->set_pktin(q_int, pktio, i); + queue_fn->set_enq_fn(q_int, pktin_enqueue); + queue_fn->set_deq_fn(q_int, pktin_dequeue); + queue_fn->set_enq_multi_fn(q_int, pktin_enq_multi); - queue_fn->set_deq_multi_fn(qentry, + queue_fn->set_deq_multi_fn(q_int, pktin_deq_multi); } @@ -1374,7 +1374,7 @@ int odp_pktout_queue_config(odp_pktio_t pktio, for (i = 0; i < num_queues; i++) { odp_queue_t queue; odp_queue_param_t queue_param; - queue_t qentry; + queue_t q_int; char name[ODP_QUEUE_NAME_LEN]; int pktio_id = pktio_to_id(pktio); @@ -1394,14 +1394,14 @@ int odp_pktout_queue_config(odp_pktio_t pktio, return -1; } - qentry = queue_fn->from_ext(queue); - queue_fn->set_pktout(qentry, pktio, i); + q_int = queue_fn->from_ext(queue); + queue_fn->set_pktout(q_int, pktio, i); /* Override default enqueue / dequeue functions */ - queue_fn->set_enq_fn(qentry, pktout_enqueue); - queue_fn->set_deq_fn(qentry, pktout_dequeue); - queue_fn->set_enq_multi_fn(qentry, pktout_enq_multi); - queue_fn->set_deq_multi_fn(qentry, pktout_deq_multi); + queue_fn->set_enq_fn(q_int, pktout_enqueue); + queue_fn->set_deq_fn(q_int, pktout_dequeue); + queue_fn->set_enq_multi_fn(q_int, pktout_enq_multi); + queue_fn->set_deq_multi_fn(q_int, pktout_deq_multi); entry->s.out_queue[i].queue = queue; } commit 91c4f94f1ed01ed22b7cd489016016b0599f4065 Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Tue Jul 4 12:53:38 2017 +0300 linux-gen: cls: avoid queue conversions Use internal queue type instead of API handle. This way per packet queue conversions are avoided. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_packet_internal.h b/platform/linux-generic/include/odp_packet_internal.h index cdd139bb..d513f05e 100644 --- a/platform/linux-generic/include/odp_packet_internal.h +++ b/platform/linux-generic/include/odp_packet_internal.h @@ -27,6 +27,7 @@ extern "C" { #include <odp/api/crypto.h> #include <odp_crypto_internal.h> #include <odp/api/plat/packet_types.h> +#include <odp_queue_if.h> /** Minimum segment length expected by packet_parse_common() */ #define PACKET_PARSE_SEG_LEN 96 @@ -126,7 +127,7 @@ typedef struct { odp_time_t timestamp; /* Classifier destination queue */ - odp_queue_t dst_queue; + queue_t dst_queue; /* Result for crypto */ odp_crypto_generic_op_result_t op_result; diff --git a/platform/linux-generic/include/odp_queue_if.h b/platform/linux-generic/include/odp_queue_if.h index 168d0e9e..b5cbd515 100644 --- a/platform/linux-generic/include/odp_queue_if.h +++ b/platform/linux-generic/include/odp_queue_if.h @@ -48,6 +48,8 @@ typedef struct { typedef struct { char dummy; } _queue_t; typedef _queue_t *queue_t; +#define QUEUE_NULL ((queue_t)NULL) + typedef int (*queue_init_global_fn_t)(void); typedef int (*queue_term_global_fn_t)(void); typedef int (*queue_init_local_fn_t)(void); diff --git a/platform/linux-generic/odp_classification.c b/platform/linux-generic/odp_classification.c index 5f153e73..cc08b9f4 100644 --- a/platform/linux-generic/odp_classification.c +++ b/platform/linux-generic/odp_classification.c @@ -162,7 +162,7 @@ odp_cos_t odp_cls_cos_create(const char *name, odp_cls_cos_param_t *param) /* Packets are dropped if Queue or Pool is invalid*/ if (param->queue == ODP_QUEUE_INVALID) - queue = NULL; + queue = QUEUE_NULL; else queue = queue_fn->from_ext(param->queue); @@ -264,7 +264,7 @@ int odp_cos_queue_set(odp_cos_t cos_id, odp_queue_t queue_id) /* Locking is not required as intermittent stale data during CoS modification is acceptable*/ if (queue_id == ODP_QUEUE_INVALID) - cos->s.queue = NULL; + cos->s.queue = QUEUE_NULL; else cos->s.queue = queue_fn->from_ext(queue_id); return 0; @@ -279,7 +279,7 @@ odp_queue_t odp_cos_queue(odp_cos_t cos_id) return ODP_QUEUE_INVALID; } - if (!cos->s.queue) + if (cos->s.queue == QUEUE_NULL) return ODP_QUEUE_INVALID; return queue_fn->to_ext(cos->s.queue); @@ -841,12 +841,12 @@ int cls_classify_packet(pktio_entry_t *entry, const uint8_t *base, if (cos == NULL) return -EINVAL; - if (cos->s.queue == NULL || cos->s.pool == ODP_POOL_INVALID) + if (cos->s.queue == QUEUE_NULL || cos->s.pool == ODP_POOL_INVALID) return -EFAULT; *pool = cos->s.pool; pkt_hdr->p.input_flags.dst_queue = 1; - pkt_hdr->dst_queue = queue_fn->to_ext(cos->s.queue); + pkt_hdr->dst_queue = cos->s.queue; return 0; } diff --git a/platform/linux-generic/odp_packet_io.c b/platform/linux-generic/odp_packet_io.c index 4dd28549..6ff35046 100644 --- a/platform/linux-generic/odp_packet_io.c +++ b/platform/linux-generic/odp_packet_io.c @@ -566,11 +566,9 @@ static inline int pktin_recv_buf(odp_pktin_queue_t queue, buf_hdr = packet_to_buf_hdr(pkt); if (pkt_hdr->p.input_flags.dst_queue) { - queue_t dst_queue; int ret; - dst_queue = queue_fn->from_ext(pkt_hdr->dst_queue); - ret = queue_fn->enq(dst_queue, buf_hdr); + ret = queue_fn->enq(pkt_hdr->dst_queue, buf_hdr); if (ret < 0) odp_packet_free(pkt); continue; commit 1cb0a8863eb8309c3e532a168fed34d6e3257218 Author: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Date: Tue Jul 4 15:54:03 2017 +0300 validation: test correctness of events subtype implementation Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Reviewed-by: Petri Savolainen <petri.savolainen(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/test/common_plat/validation/api/buffer/buffer.c b/test/common_plat/validation/api/buffer/buffer.c index 7c723d4f..66d00ab7 100644 --- a/test/common_plat/validation/api/buffer/buffer.c +++ b/test/common_plat/validation/api/buffer/buffer.c @@ -48,7 +48,8 @@ void buffer_test_pool_alloc(void) odp_buffer_t buffer[num]; odp_event_t ev; int index; - char wrong_type = 0, wrong_size = 0, wrong_align = 0; + odp_bool_t wrong_type = false, wrong_subtype = false; + odp_bool_t wrong_size = false, wrong_align = false; odp_pool_param_t params; odp_pool_param_init(&params); @@ -63,6 +64,7 @@ void buffer_test_pool_alloc(void) /* Try to allocate num items from the pool */ for (index = 0; index < num; index++) { uintptr_t addr; + odp_event_subtype_t subtype; buffer[index] = odp_buffer_alloc(pool); @@ -71,14 +73,20 @@ void buffer_test_pool_alloc(void) ev = odp_buffer_to_event(buffer[index]); if (odp_event_type(ev) != ODP_EVENT_BUFFER) - wrong_type = 1; + wrong_type = true; + if (odp_event_subtype(ev) != ODP_EVENT_NO_SUBTYPE) + wrong_subtype = true; + if (odp_event_types(ev, &subtype) != ODP_EVENT_BUFFER) + wrong_type = true; + if (subtype != ODP_EVENT_NO_SUBTYPE) + wrong_subtype = true; if (odp_buffer_size(buffer[index]) < BUF_SIZE) - wrong_size = 1; + wrong_size = true; addr = (uintptr_t)odp_buffer_addr(buffer[index]); if ((addr % BUF_ALIGN) != 0) - wrong_align = 1; + wrong_align = true; if (wrong_type || wrong_size || wrong_align) odp_buffer_print(buffer[index]); @@ -90,9 +98,10 @@ void buffer_test_pool_alloc(void) index--; /* Check that the pool had correct buffers */ - CU_ASSERT(wrong_type == 0); - CU_ASSERT(wrong_size == 0); - CU_ASSERT(wrong_align == 0); + CU_ASSERT(!wrong_type); + CU_ASSERT(!wrong_subtype); + CU_ASSERT(!wrong_size); + CU_ASSERT(!wrong_align); for (; index >= 0; index--) odp_buffer_free(buffer[index]); @@ -123,7 +132,8 @@ void buffer_test_pool_alloc_multi(void) odp_buffer_t buffer[num + 1]; odp_event_t ev; int index; - char wrong_type = 0, wrong_size = 0, wrong_align = 0; + odp_bool_t wrong_type = false, wrong_subtype = false; + odp_bool_t wrong_size = false, wrong_align = false; odp_pool_param_t params; odp_pool_param_init(&params); @@ -140,20 +150,27 @@ void buffer_test_pool_alloc_multi(void) for (index = 0; index < num; index++) { uintptr_t addr; + odp_event_subtype_t subtype; if (buffer[index] == ODP_BUFFER_INVALID) break; ev = odp_buffer_to_event(buffer[index]); if (odp_event_type(ev) != ODP_EVENT_BUFFER) - wrong_type = 1; + wrong_type = true; + if (odp_event_subtype(ev) != ODP_EVENT_NO_SUBTYPE) + wrong_subtype = true; + if (odp_event_types(ev, &subtype) != ODP_EVENT_BUFFER) + wrong_type = true; + if (subtype != ODP_EVENT_NO_SUBTYPE) + wrong_subtype = true; if (odp_buffer_size(buffer[index]) < BUF_SIZE) - wrong_size = 1; + wrong_size = true; addr = (uintptr_t)odp_buffer_addr(buffer[index]); if ((addr % BUF_ALIGN) != 0) - wrong_align = 1; + wrong_align = true; if (wrong_type || wrong_size || wrong_align) odp_buffer_print(buffer[index]); @@ -163,9 +180,10 @@ void buffer_test_pool_alloc_multi(void) CU_ASSERT(index == num); /* Check that the pool had correct buffers */ - CU_ASSERT(wrong_type == 0); - CU_ASSERT(wrong_size == 0); - CU_ASSERT(wrong_align == 0); + CU_ASSERT(!wrong_type); + CU_ASSERT(!wrong_subtype); + CU_ASSERT(!wrong_size); + CU_ASSERT(!wrong_align); odp_buffer_free_multi(buffer, num); @@ -244,10 +262,14 @@ void buffer_test_pool_free_multi(void) void buffer_test_management_basic(void) { odp_event_t ev = odp_buffer_to_event(raw_buffer); + odp_event_subtype_t subtype; CU_ASSERT(odp_buffer_is_valid(raw_buffer) == 1); CU_ASSERT(odp_buffer_pool(raw_buffer) != ODP_POOL_INVALID); CU_ASSERT(odp_event_type(ev) == ODP_EVENT_BUFFER); + CU_ASSERT(odp_event_subtype(ev) == ODP_EVENT_NO_SUBTYPE); + CU_ASSERT(odp_event_types(ev, &subtype) == ODP_EVENT_BUFFER); + CU_ASSERT(subtype == ODP_EVENT_NO_SUBTYPE); CU_ASSERT(odp_buffer_size(raw_buffer) >= BUF_SIZE); CU_ASSERT(odp_buffer_addr(raw_buffer) != NULL); odp_buffer_print(raw_buffer); diff --git a/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c b/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c index ae600e23..1d7d49a2 100644 --- a/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c +++ b/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c @@ -103,6 +103,7 @@ static void alg_test(odp_crypto_op_t op, odp_crypto_auth_capability_t auth_capa[MAX_ALG_CAPA]; int num, i; int found; + odp_event_subtype_t subtype; rc = odp_crypto_capability(&capa); CU_ASSERT(!rc); @@ -263,6 +264,12 @@ static void alg_test(odp_crypto_op_t op, event = odp_queue_deq(suite_context.queue); } while (event == ODP_EVENT_INVALID); + CU_ASSERT(ODP_EVENT_CRYPTO_COMPL == odp_event_type(event)); + CU_ASSERT(ODP_EVENT_NO_SUBTYPE == odp_event_subtype(event)); + CU_ASSERT(ODP_EVENT_CRYPTO_COMPL == + odp_event_types(event, &subtype)); + CU_ASSERT(ODP_EVENT_NO_SUBTYPE == subtype); + compl_event = odp_crypto_compl_from_event(event); CU_ASSERT(odp_crypto_compl_to_u64(compl_event) == odp_crypto_compl_to_u64(odp_crypto_compl_from_event(event))); @@ -272,6 +279,13 @@ static void alg_test(odp_crypto_op_t op, CU_ASSERT(result.pkt == pkt); CU_ASSERT(result.ctx == (void *)0xdeadbeef); + CU_ASSERT(ODP_EVENT_PACKET == + odp_event_type(odp_packet_to_event(result.pkt))); + CU_ASSERT(ODP_EVENT_PACKET_BASIC == + odp_event_subtype(odp_packet_to_event(result.pkt))); + CU_ASSERT(ODP_EVENT_PACKET == + odp_event_types(odp_packet_to_event(result.pkt), &subtype)); + CU_ASSERT(ODP_EVENT_PACKET_BASIC == subtype); if (should_fail) { CU_ASSERT(!result.ok); diff --git a/test/common_plat/validation/api/packet/packet.c b/test/common_plat/validation/api/packet/packet.c index 284aaeb5..fa6f4afd 100644 --- a/test/common_plat/validation/api/packet/packet.c +++ b/test/common_plat/validation/api/packet/packet.c @@ -241,6 +241,7 @@ void packet_test_alloc_free(void) odp_packet_t packet; odp_pool_param_t params; odp_pool_capability_t capa; + odp_event_subtype_t subtype; CU_ASSERT_FATAL(odp_pool_capability(&capa) == 0); @@ -259,7 +260,12 @@ void packet_test_alloc_free(void) CU_ASSERT_FATAL(packet != ODP_PACKET_INVALID); CU_ASSERT(odp_packet_len(packet) == packet_len); CU_ASSERT(odp_event_type(odp_packet_to_event(packet)) == - ODP_EVENT_PACKET); + ODP_EVENT_PACKET); + CU_ASSERT(odp_event_subtype(odp_packet_to_event(packet)) == + ODP_EVENT_PACKET_BASIC); + CU_ASSERT(odp_event_types(odp_packet_to_event(packet), &subtype) == + ODP_EVENT_PACKET); + CU_ASSERT(subtype == ODP_EVENT_PACKET_BASIC); CU_ASSERT(odp_packet_to_u64(packet) != odp_packet_to_u64(ODP_PACKET_INVALID)); @@ -329,9 +335,17 @@ void packet_test_alloc_free_multi(void) CU_ASSERT_FATAL(ret == num_pkt); for (i = 0; i < 2 * num_pkt; ++i) { + odp_event_subtype_t subtype; + CU_ASSERT(odp_packet_len(packet[i]) == packet_len); CU_ASSERT(odp_event_type(odp_packet_to_event(packet[i])) == ODP_EVENT_PACKET); + CU_ASSERT(odp_event_subtype(odp_packet_to_event(packet[i])) == + ODP_EVENT_PACKET_BASIC); + CU_ASSERT(odp_event_types(odp_packet_to_event(packet[i]), + &subtype) == + ODP_EVENT_PACKET); + CU_ASSERT(subtype == ODP_EVENT_PACKET_BASIC); CU_ASSERT(odp_packet_to_u64(packet[i]) != odp_packet_to_u64(ODP_PACKET_INVALID)); } @@ -449,10 +463,15 @@ void packet_test_event_conversion(void) odp_packet_t pkt = test_packet; odp_packet_t tmp_pkt; odp_event_t ev; + odp_event_subtype_t subtype; ev = odp_packet_to_event(pkt); CU_ASSERT_FATAL(ev != ODP_EVENT_INVALID); CU_ASSERT(odp_event_type(ev) == ODP_EVENT_PACKET); + CU_ASSERT(odp_event_subtype(ev) == ODP_EVENT_PACKET_BASIC); + CU_ASSERT(odp_event_types(ev, &subtype) == + ODP_EVENT_PACKET); + CU_ASSERT(subtype == ODP_EVENT_PACKET_BASIC); tmp_pkt = odp_packet_from_event(ev); CU_ASSERT_FATAL(tmp_pkt != ODP_PACKET_INVALID); diff --git a/test/common_plat/validation/api/timer/timer.c b/test/common_plat/validation/api/timer/timer.c index b7d84c64..88135160 100644 --- a/test/common_plat/validation/api/timer/timer.c +++ b/test/common_plat/validation/api/timer/timer.c @@ -59,7 +59,7 @@ void timer_test_timeout_pool_alloc(void) odp_timeout_t tmo[num]; odp_event_t ev; int index; - char wrong_type = 0; + odp_bool_t wrong_type = false, wrong_subtype = false; odp_pool_param_t params; odp_pool_param_init(&params); @@ -73,6 +73,8 @@ void timer_test_timeout_pool_alloc(void) /* Try to allocate num items from the pool */ for (index = 0; index < num; index++) { + odp_event_subtype_t subtype; + tmo[index] = odp_timeout_alloc(pool); if (tmo[index] == ODP_TIMEOUT_INVALID) @@ -80,7 +82,13 @@ void timer_test_timeout_pool_alloc(void) ev = odp_timeout_to_event(tmo[index]); if (odp_event_type(ev) != ODP_EVENT_TIMEOUT) - wrong_type = 1; + wrong_type = true; + if (odp_event_subtype(ev) != ODP_EVENT_NO_SUBTYPE) + wrong_subtype = true; + if (odp_event_types(ev, &subtype) != ODP_EVENT_TIMEOUT) + wrong_type = true; + if (subtype != ODP_EVENT_NO_SUBTYPE) + wrong_subtype = true; } /* Check that the pool had at least num items */ @@ -89,7 +97,8 @@ void timer_test_timeout_pool_alloc(void) index--; /* Check that the pool had correct buffers */ - CU_ASSERT(wrong_type == 0); + CU_ASSERT(!wrong_type); + CU_ASSERT(!wrong_subtype); for (; index >= 0; index--) odp_timeout_free(tmo[index]); @@ -219,12 +228,29 @@ void timer_test_odp_timer_cancel(void) /* @private Handle a received (timeout) event */ static void handle_tmo(odp_event_t ev, bool stale, uint64_t prev_tick) { + odp_event_subtype_t subtype; + CU_ASSERT_FATAL(ev != ODP_EVENT_INVALID); /* Internal error */ if (odp_event_type(ev) != ODP_EVENT_TIMEOUT) { /* Not a timeout event */ CU_FAIL("Unexpected event type received"); return; } + if (odp_event_subtype(ev) != ODP_EVENT_NO_SUBTYPE) { + /* Not a timeout event */ + CU_FAIL("Unexpected event subtype received"); + return; + } + if (odp_event_types(ev, &subtype) != ODP_EVENT_TIMEOUT) { + /* Not a timeout event */ + CU_FAIL("Unexpected event type received"); + return; + } + if (subtype != ODP_EVENT_NO_SUBTYPE) { + /* Not a timeout event */ + CU_FAIL("Unexpected event subtype received"); + return; + } /* Read the metadata from the timeout */ odp_timeout_t tmo = odp_timeout_from_event(ev); odp_timer_t tim = odp_timeout_timer(tmo); commit faf97a341df8e92fd692029fe2c6c353f043950b Author: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Date: Tue Jul 4 15:54:02 2017 +0300 linux-generic: events subtype implementation Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov(a)linaro.org> Reviewed-by: Petri Savolainen <petri.savolainen(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_buffer_inlines.h b/platform/linux-generic/include/odp_buffer_inlines.h index cf817d90..4c0e7339 100644 --- a/platform/linux-generic/include/odp_buffer_inlines.h +++ b/platform/linux-generic/include/odp_buffer_inlines.h @@ -21,6 +21,8 @@ extern "C" { odp_event_type_t _odp_buffer_event_type(odp_buffer_t buf); void _odp_buffer_event_type_set(odp_buffer_t buf, int ev); +odp_event_subtype_t _odp_buffer_event_subtype(odp_buffer_t buf); +void _odp_buffer_event_subtype_set(odp_buffer_t buf, int ev); int odp_buffer_snprint(char *str, uint32_t n, odp_buffer_t buf); static inline odp_buffer_t odp_hdr_to_buf(odp_buffer_hdr_t *hdr) diff --git a/platform/linux-generic/include/odp_buffer_internal.h b/platform/linux-generic/include/odp_buffer_internal.h index 076abe96..dadf285e 100644 --- a/platform/linux-generic/include/odp_buffer_internal.h +++ b/platform/linux-generic/include/odp_buffer_internal.h @@ -96,6 +96,9 @@ struct odp_buffer_hdr_t { /* Event type. Maybe different than pool type (crypto compl event) */ int8_t event_type; + /* Event subtype. Should be ODP_EVENT_NO_SUBTYPE except packets. */ + int8_t event_subtype; + /* Burst table */ struct odp_buffer_hdr_t *burst[BUFFER_BURST_SIZE]; diff --git a/platform/linux-generic/odp_crypto.c b/platform/linux-generic/odp_crypto.c index 68fc5658..0e2adad1 100644 --- a/platform/linux-generic/odp_crypto.c +++ b/platform/linux-generic/odp_crypto.c @@ -907,12 +907,13 @@ odp_crypto_operation(odp_crypto_op_param_t *param, if (ODP_QUEUE_INVALID != session->p.compl_queue) { odp_event_t completion_event; odp_crypto_generic_op_result_t *op_result; + odp_buffer_t buf; /* Linux generic will always use packet for completion event */ completion_event = odp_packet_to_event(param->out_pkt); - _odp_buffer_event_type_set( - odp_buffer_from_event(completion_event), - ODP_EVENT_CRYPTO_COMPL); + buf = odp_buffer_from_event(completion_event); + _odp_buffer_event_type_set(buf, ODP_EVENT_CRYPTO_COMPL); + _odp_buffer_event_subtype_set(buf, ODP_EVENT_NO_SUBTYPE); /* Asynchronous, build result (no HW so no errors) and send it*/ op_result = get_op_result_from_event(completion_event); op_result->magic = OP_RESULT_MAGIC; @@ -1106,9 +1107,11 @@ odp_crypto_compl_result(odp_crypto_compl_t completion_event, void odp_crypto_compl_free(odp_crypto_compl_t completion_event) { - _odp_buffer_event_type_set( - odp_buffer_from_event((odp_event_t)completion_event), - ODP_EVENT_PACKET); + odp_buffer_t buf = + odp_buffer_from_event((odp_event_t)completion_event); + + _odp_buffer_event_type_set(buf, ODP_EVENT_PACKET); + _odp_buffer_event_subtype_set(buf, ODP_EVENT_PACKET_BASIC); } void odp_crypto_session_param_init(odp_crypto_session_param_t *param) diff --git a/platform/linux-generic/odp_event.c b/platform/linux-generic/odp_event.c index d71f4464..23b410d8 100644 --- a/platform/linux-generic/odp_event.c +++ b/platform/linux-generic/odp_event.c @@ -19,6 +19,21 @@ odp_event_type_t odp_event_type(odp_event_t event) return _odp_buffer_event_type(odp_buffer_from_event(event)); } +odp_event_subtype_t odp_event_subtype(odp_event_t event) +{ + return _odp_buffer_event_subtype(odp_buffer_from_event(event)); +} + +odp_event_type_t odp_event_types(odp_event_t event, + odp_event_subtype_t *subtype) +{ + odp_buffer_t buf = odp_buffer_from_event(event); + + *subtype = _odp_buffer_event_subtype(buf); + + return _odp_buffer_event_type(buf); +} + void odp_event_free(odp_event_t event) { switch (odp_event_type(event)) { diff --git a/platform/linux-generic/odp_packet.c b/platform/linux-generic/odp_packet.c index eb66af2d..3789feca 100644 --- a/platform/linux-generic/odp_packet.c +++ b/platform/linux-generic/odp_packet.c @@ -268,6 +268,7 @@ static inline void packet_init(odp_packet_hdr_t *pkt_hdr, uint32_t len) CONFIG_PACKET_TAILROOM; pkt_hdr->input = ODP_PKTIO_INVALID; + pkt_hdr->buf_hdr.event_subtype = ODP_EVENT_PACKET_BASIC; } static inline void init_segments(odp_packet_hdr_t *pkt_hdr[], int num) diff --git a/platform/linux-generic/odp_pool.c b/platform/linux-generic/odp_pool.c index 9dba7341..23b80698 100644 --- a/platform/linux-generic/odp_pool.c +++ b/platform/linux-generic/odp_pool.c @@ -259,6 +259,7 @@ static void init_buffers(pool_t *pool) buf_hdr->size = seg_size; buf_hdr->type = type; buf_hdr->event_type = type; + buf_hdr->event_subtype = ODP_EVENT_NO_SUBTYPE; buf_hdr->pool_hdl = pool->pool_hdl; buf_hdr->uarea_addr = uarea; /* Show user requested size through API */ @@ -566,6 +567,16 @@ void _odp_buffer_event_type_set(odp_buffer_t buf, int ev) buf_hdl_to_hdr(buf)->event_type = ev; } +odp_event_subtype_t _odp_buffer_event_subtype(odp_buffer_t buf) +{ + return buf_hdl_to_hdr(buf)->event_subtype; +} + +void _odp_buffer_event_subtype_set(odp_buffer_t buf, int ev) +{ + buf_hdl_to_hdr(buf)->event_subtype = ev; +} + odp_pool_t odp_pool_lookup(const char *name) { uint32_t i; ----------------------------------------------------------------------- Summary of changes: platform/linux-generic/Makefile.am | 1 - .../linux-generic/include/odp_buffer_inlines.h | 2 + .../linux-generic/include/odp_buffer_internal.h | 3 + .../linux-generic/include/odp_packet_internal.h | 3 +- .../linux-generic/include/odp_packet_io_internal.h | 2 + .../linux-generic/include/odp_packet_io_queue.h | 48 -------- platform/linux-generic/include/odp_queue_if.h | 20 ++- platform/linux-generic/odp_classification.c | 10 +- platform/linux-generic/odp_crypto.c | 15 ++- platform/linux-generic/odp_event.c | 15 +++ platform/linux-generic/odp_packet.c | 1 + platform/linux-generic/odp_packet_io.c | 137 +++++++++++---------- platform/linux-generic/odp_pool.c | 11 ++ platform/linux-generic/odp_queue.c | 39 +++--- platform/linux-generic/odp_queue_scalable.c | 39 +++--- platform/linux-generic/odp_traffic_mngr.c | 10 +- test/common_plat/validation/api/buffer/buffer.c | 50 +++++--- .../validation/api/crypto/odp_crypto_test_inp.c | 14 +++ test/common_plat/validation/api/packet/packet.c | 21 +++- test/common_plat/validation/api/timer/timer.c | 32 ++++- 20 files changed, 258 insertions(+), 215 deletions(-) delete mode 100644 platform/linux-generic/include/odp_packet_io_queue.h hooks/post-receive --

9 years, 1 month

[COMMIT] branch master updated. v1.15.0.0-34-g29641a1e

by git＠git-us.linaro.org

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 29641a1e0d0868c09f35f9f32e6c8491efb62b2d (commit) via cfaee18006aab817656ffaf09e644b570ff32c8e (commit) via b02fe5f06a81a190d4c4fe65af74b800cf073436 (commit) from 5a2ed3eefb1a263139282c376741adb2c355bd5a (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 29641a1e0d0868c09f35f9f32e6c8491efb62b2d Author: Maxim Uvarov <maxim.uvarov(a)linaro.org> Date: Mon Jul 10 13:37:28 2017 +0300 test: pktio: ipc: increse timeouts for coverage test Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> diff --git a/test/linux-generic/pktio_ipc/pktio_ipc_run.sh b/test/linux-generic/pktio_ipc/pktio_ipc_run.sh index c53de457..b713c63c 100755 --- a/test/linux-generic/pktio_ipc/pktio_ipc_run.sh +++ b/test/linux-generic/pktio_ipc/pktio_ipc_run.sh @@ -17,20 +17,29 @@ PATH=$(dirname $0):$PATH PATH=$(dirname $0)/../../../../platform/linux-generic/test/pktio_ipc:$PATH PATH=.:$PATH +RUNTIME1=10 +RUNTIME2=5 +TIMEOUT=13 +if [ "${TEST}" = "coverage" ]; then + RUNTIME1=30 + RUNTIME2=15 + TIMEOUT=20 +fi + run() { local ret=0 echo "==== run pktio_ipc1 then pktio_ipc2 ====" - pktio_ipc1${EXEEXT} -t 10 & + pktio_ipc1${EXEEXT} -t ${RUNTIME1} & IPC_PID=$! - pktio_ipc2${EXEEXT} -p ${IPC_PID} -t 5 + pktio_ipc2${EXEEXT} -p ${IPC_PID} -t ${RUNTIME2} ret=$? # pktio_ipc1 should do clean up and exit just # after pktio_ipc2 exited. If it does not happen # kill him in test. - sleep 13 + sleep ${TIMEOUT} (kill ${IPC_PID} 2>&1 > /dev/null ) > /dev/null if [ $? -eq 0 ]; then echo "pktio_ipc1${EXEEXT} was killed" @@ -49,14 +58,14 @@ run() fi echo "==== run pktio_ipc2 then pktio_ipc1 ====" - pktio_ipc2${EXEEXT} -t 10 & + pktio_ipc2${EXEEXT} -t ${RUNTIME1} & IPC_PID=$! - pktio_ipc1${EXEEXT} -p ${IPC_PID} -t 5 + pktio_ipc1${EXEEXT} -p ${IPC_PID} -t ${RUNTIME2} ret=$? # pktio_ipc2 do not exit on pktio_ipc1 disconnect # wait until it exits cleanly - sleep 13 + sleep ${TIMEOUT} (kill ${IPC_PID} 2>&1 > /dev/null ) > /dev/null if [ $? -eq 0 ]; then echo "pktio_ipc2${EXEEXT} was killed" commit cfaee18006aab817656ffaf09e644b570ff32c8e Author: Maxim Uvarov <maxim.uvarov(a)linaro.org> Date: Mon Jul 10 13:27:14 2017 +0300 test: ipc pktio use /dev/shm instead of tmp Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> diff --git a/test/linux-generic/pktio_ipc/pktio_ipc_run.sh b/test/linux-generic/pktio_ipc/pktio_ipc_run.sh index 331ecdc9..c53de457 100755 --- a/test/linux-generic/pktio_ipc/pktio_ipc_run.sh +++ b/test/linux-generic/pktio_ipc/pktio_ipc_run.sh @@ -34,11 +34,11 @@ run() (kill ${IPC_PID} 2>&1 > /dev/null ) > /dev/null if [ $? -eq 0 ]; then echo "pktio_ipc1${EXEEXT} was killed" - ls -l /tmp/odp* 2> /dev/null - rm -rf /tmp/odp-${IPC_PID}* 2>&1 > /dev/null + ls -l /dev/shm/${UID}/odp* 2> /dev/null + rm -rf /dev/shm/${UID}/odp-${IPC_PID}* 2>&1 > /dev/null else echo "normal exit of 2 application" - ls -l /tmp/odp* 2> /dev/null + ls -l /dev/shm/${UID}/odp* 2> /dev/null fi if [ $ret -ne 0 ]; then @@ -60,20 +60,20 @@ run() (kill ${IPC_PID} 2>&1 > /dev/null ) > /dev/null if [ $? -eq 0 ]; then echo "pktio_ipc2${EXEEXT} was killed" - ls -l /tmp/odp* 2> /dev/null - rm -rf /tmp/odp-${IPC_PID}* 2>&1 > /dev/null + ls -l /dev/shm/${UID}/odp* 2> /dev/null + rm -rf /dev/shm/${UID}/odp-${IPC_PID}* 2>&1 > /dev/null else echo "normal exit of 2 application" - ls -l /tmp/odp* 2> /dev/null + ls -l /dev/shm/${UID}/odp* 2> /dev/null fi if [ $ret -ne 0 ]; then echo "!!! FAILED !!!" - ls -l /tmp/odp* 2> /dev/null - rm -rf /tmp/odp-${IPC_PID}* 2>&1 > /dev/null + ls -l /dev/shm/${UID}/odp* 2> /dev/null + rm -rf /dev/shm/${UID}/odp-${IPC_PID}* 2>&1 > /dev/null exit $ret else - ls -l /tmp/odp* 2> /dev/null + ls -l /dev/shm/${UID}/odp* 2> /dev/null echo "Second stage PASSED" fi commit b02fe5f06a81a190d4c4fe65af74b800cf073436 Author: Maxim Uvarov <maxim.uvarov(a)linaro.org> Date: Sat Jul 8 00:50:54 2017 +0300 linux-gen: pktio: ipc fix packets free Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> diff --git a/platform/linux-generic/pktio/ipc.c b/platform/linux-generic/pktio/ipc.c index 54e42227..f0facd8c 100644 --- a/platform/linux-generic/pktio/ipc.c +++ b/platform/linux-generic/pktio/ipc.c @@ -389,10 +389,15 @@ static void _ipc_free_ring_packets(pktio_entry_t *pktio_entry, _ring_t *r) int ret; void **rbuf_p; int i; + void *addr; + pool_t *pool; if (!r) return; + pool = pool_entry_from_hdl(pktio_entry->s.ipc.pool); + addr = odp_shm_addr(pool->shm); + rbuf_p = (void *)&offsets; while (1) { @@ -403,10 +408,10 @@ static void _ipc_free_ring_packets(pktio_entry_t *pktio_entry, _ring_t *r) for (i = 0; i < ret; i++) { odp_packet_hdr_t *phdr; odp_packet_t pkt; - void *mbase = pktio_entry->s.ipc.pool_mdata_base; - phdr = (void *)((uint8_t *)mbase + offsets[i]); + phdr = (void *)((uint8_t *)addr + offsets[i]); pkt = packet_handle(phdr); + odp_packet_free(pkt); } } @@ -420,7 +425,7 @@ static int ipc_pktio_recv_lockless(pktio_entry_t *pktio_entry, _ring_t *r; _ring_t *r_p; uintptr_t offsets[PKTIO_IPC_ENTRIES]; - void **ipcbufs_p = (void *)&offsets; + void **ipcbufs_p = (void *)&offsets[0]; uint32_t ready; int pkts_ring; @@ -437,6 +442,11 @@ static int ipc_pktio_recv_lockless(pktio_entry_t *pktio_entry, if (odp_unlikely(pkts < 0)) ODP_ABORT("internal error dequeue\n"); + for (i = 0; i < pkts; i++) { + IPC_ODP_DBG("%d/%d recv packet offset %x\n", + i, pkts, offsets[i]); + } + /* fast path */ if (odp_likely(0 == pkts)) return 0; @@ -511,9 +521,17 @@ static int ipc_pktio_recv_lockless(pktio_entry_t *pktio_entry, r_p = pktio_entry->s.ipc.rx.free; repeat: + + ipcbufs_p = (void *)&offsets[0]; pkts_ring = _ring_mp_enqueue_burst(r_p, ipcbufs_p, pkts); if (odp_unlikely(pkts_ring < 0)) ODP_ABORT("ipc: odp_ring_mp_enqueue_bulk r_p fail\n"); + + for (i = 0; i < pkts; i++) { + IPC_ODP_DBG("%d/%d send to be free packet offset %x\n", + i, pkts, offsets[i]); + } + if (odp_unlikely(pkts != pkts_ring)) { IPC_ODP_DBG("odp_ring_full: %d, odp_ring_count %d," " _ring_free_count %d\n", @@ -597,10 +615,13 @@ static int ipc_pktio_send_lockless(pktio_entry_t *pktio_entry, /* compile all function code even if ipc disabled with config */ pkt_hdr->buf_hdr.ipc_data_offset = data_pool_off; IPC_ODP_DBG("%d/%d send packet %llx, pool %llx," - "phdr = %p, offset %x\n", + "phdr = %p, offset %x sendoff %x, addr %llx iaddr %llx\n", i, len, odp_packet_to_u64(pkt), odp_pool_to_u64(pool_hdl), - pkt_hdr, pkt_hdr->buf_hdr.ipc_data_offset); + pkt_hdr, pkt_hdr->buf_hdr.ipc_data_offset, + offsets[i], odp_shm_addr(pool->shm), + odp_shm_addr(pool_entry_from_hdl( + pktio_entry->s.ipc.pool)->shm)); } /* Put packets to ring to be processed by other process. */ ----------------------------------------------------------------------- Summary of changes: platform/linux-generic/pktio/ipc.c | 31 +++++++++++++++++---- test/linux-generic/pktio_ipc/pktio_ipc_run.sh | 39 ++++++++++++++++----------- 2 files changed, 50 insertions(+), 20 deletions(-) hooks/post-receive --

9 years, 1 month

[COMMIT] branch api-next updated. v1.15.0.0-403-gc7718962

by git＠git-us.linaro.org

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, api-next has been updated via c7718962c6633c80eb71a0400d89c31f11f88045 (commit) via 79ce62cf3266b0d1b05acc68be382cd6271b7371 (commit) via 39e4ebd39f2d171bbfef5bfee6596033b83c56a0 (commit) via f9e7f458cf7151c75a9145272c62306b96f9ff27 (commit) from b668182d6ea0cb942c2cf43771c618c9457bf146 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit c7718962c6633c80eb71a0400d89c31f11f88045 Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Fri Jun 30 17:10:56 2017 +0300 linux-gen: sched: remove unused sched interface functions Removed functions that are no longer used. Also removed unused parameter from save_context function. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_schedule_if.h b/platform/linux-generic/include/odp_schedule_if.h index 5abbb732..b514c88a 100644 --- a/platform/linux-generic/include/odp_schedule_if.h +++ b/platform/linux-generic/include/odp_schedule_if.h @@ -35,7 +35,7 @@ typedef int (*schedule_term_local_fn_t)(void); typedef void (*schedule_order_lock_fn_t)(void); typedef void (*schedule_order_unlock_fn_t)(void); typedef unsigned (*schedule_max_ordered_locks_fn_t)(void); -typedef void (*schedule_save_context_fn_t)(uint32_t queue_index, void *ptr); +typedef void (*schedule_save_context_fn_t)(uint32_t queue_index); typedef struct schedule_fn_t { int status_sync; @@ -68,11 +68,6 @@ extern const schedule_fn_t *sched_fn; int sched_cb_pktin_poll(int pktio_index, int num_queue, int index[]); void sched_cb_pktio_stop_finalize(int pktio_index); int sched_cb_num_pktio(void); -int sched_cb_num_queues(void); -int sched_cb_queue_prio(uint32_t queue_index); -int sched_cb_queue_grp(uint32_t queue_index); -int sched_cb_queue_is_ordered(uint32_t queue_index); -int sched_cb_queue_is_atomic(uint32_t queue_index); odp_queue_t sched_cb_queue_handle(uint32_t queue_index); void sched_cb_queue_destroy_finalize(uint32_t queue_index); int sched_cb_queue_deq_multi(uint32_t queue_index, odp_event_t ev[], int num); diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c index d907779b..4c85027b 100644 --- a/platform/linux-generic/odp_queue.c +++ b/platform/linux-generic/odp_queue.c @@ -520,7 +520,7 @@ static inline int deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], queue->s.tail = NULL; if (status_sync && queue->s.type == ODP_QUEUE_TYPE_SCHED) - sched_fn->save_context(queue->s.index, queue); + sched_fn->save_context(queue->s.index); UNLOCK(&queue->s.lock); @@ -672,25 +672,6 @@ static int queue_info(odp_queue_t handle, odp_queue_info_t *info) return 0; } -int sched_cb_num_queues(void) -{ - return ODP_CONFIG_QUEUES; -} - -int sched_cb_queue_prio(uint32_t queue_index) -{ - queue_entry_t *qe = get_qentry(queue_index); - - return qe->s.param.sched.prio; -} - -int sched_cb_queue_grp(uint32_t queue_index) -{ - queue_entry_t *qe = get_qentry(queue_index); - - return qe->s.param.sched.group; -} - odp_queue_t sched_cb_queue_handle(uint32_t queue_index) { return queue_from_id(queue_index); diff --git a/platform/linux-generic/odp_schedule_iquery.c b/platform/linux-generic/odp_schedule_iquery.c index f315a4f0..9605edc7 100644 --- a/platform/linux-generic/odp_schedule_iquery.c +++ b/platform/linux-generic/odp_schedule_iquery.c @@ -1308,10 +1308,8 @@ static inline bool is_ordered_queue(unsigned int queue_index) return (sched->queues[queue_index].sync == ODP_SCHED_SYNC_ORDERED); } -static void schedule_save_context(uint32_t queue_index, void *ptr) +static void schedule_save_context(uint32_t queue_index) { - (void)ptr; - if (is_atomic_queue(queue_index)) { thread_local.atomic = &sched->availables[queue_index]; } else if (is_ordered_queue(queue_index)) { commit 79ce62cf3266b0d1b05acc68be382cd6271b7371 Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Fri Jun 30 17:10:55 2017 +0300 linux-gen: sched: remove most dependecies to qentry Moved ordered queue context structure from queue internal structure to the scheduler. Ordering is a scheduler feature and thus all data and code about ordering should be in scheduler implementation. This removes most dependencies to qentry from the scheduler. Remaining dependecies are due to queue interface definition, which is not changed in this patch. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_queue_internal.h b/platform/linux-generic/include/odp_queue_internal.h index d79abd23..032dde88 100644 --- a/platform/linux-generic/include/odp_queue_internal.h +++ b/platform/linux-generic/include/odp_queue_internal.h @@ -42,13 +42,6 @@ struct queue_entry_s { odp_buffer_hdr_t *tail; int status; - struct { - odp_atomic_u64_t ctx; /**< Current ordered context id */ - odp_atomic_u64_t next_ctx; /**< Next unallocated context id */ - /** Array of ordered locks */ - odp_atomic_u64_t lock[CONFIG_QUEUE_MAX_ORD_LOCKS]; - } ordered ODP_ALIGNED_CACHE; - queue_enq_fn_t enqueue ODP_ALIGNED_CACHE; queue_deq_fn_t dequeue; queue_enq_multi_fn_t enqueue_multi; diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c index 2db95fc6..d907779b 100644 --- a/platform/linux-generic/odp_queue.c +++ b/platform/linux-generic/odp_queue.c @@ -57,16 +57,6 @@ static inline odp_queue_t queue_from_id(uint32_t queue_id) return _odp_cast_scalar(odp_queue_t, queue_id + 1); } -static inline int queue_is_atomic(queue_entry_t *qe) -{ - return qe->s.param.sched.sync == ODP_SCHED_SYNC_ATOMIC; -} - -static inline int queue_is_ordered(queue_entry_t *qe) -{ - return qe->s.param.sched.sync == ODP_SCHED_SYNC_ORDERED; -} - queue_entry_t *get_qentry(uint32_t queue_id) { return &queue_tbl->queue[queue_id]; @@ -278,13 +268,6 @@ static int queue_destroy(odp_queue_t handle) ODP_ERR("queue \"%s\" not empty\n", queue->s.name); return -1; } - if (queue_is_ordered(queue) && - odp_atomic_load_u64(&queue->s.ordered.ctx) != - odp_atomic_load_u64(&queue->s.ordered.next_ctx)) { - UNLOCK(&queue->s.lock); - ODP_ERR("queue \"%s\" reorder incomplete\n", queue->s.name); - return -1; - } switch (queue->s.status) { case QUEUE_STATUS_READY: @@ -610,20 +593,9 @@ static int queue_init(queue_entry_t *queue, const char *name, if (queue->s.param.sched.lock_count > sched_fn->max_ordered_locks()) return -1; - if (param->type == ODP_QUEUE_TYPE_SCHED) { + if (param->type == ODP_QUEUE_TYPE_SCHED) queue->s.param.deq_mode = ODP_QUEUE_OP_DISABLED; - if (param->sched.sync == ODP_SCHED_SYNC_ORDERED) { - unsigned i; - - odp_atomic_init_u64(&queue->s.ordered.ctx, 0); - odp_atomic_init_u64(&queue->s.ordered.next_ctx, 0); - - for (i = 0; i < queue->s.param.sched.lock_count; i++) - odp_atomic_init_u64(&queue->s.ordered.lock[i], - 0); - } - } queue->s.type = queue->s.param.type; queue->s.enqueue = queue_int_enq; @@ -719,16 +691,6 @@ int sched_cb_queue_grp(uint32_t queue_index) return qe->s.param.sched.group; } -int sched_cb_queue_is_ordered(uint32_t queue_index) -{ - return queue_is_ordered(get_qentry(queue_index)); -} - -int sched_cb_queue_is_atomic(uint32_t queue_index) -{ - return queue_is_atomic(get_qentry(queue_index)); -} - odp_queue_t sched_cb_queue_handle(uint32_t queue_index) { return queue_from_id(queue_index); diff --git a/platform/linux-generic/odp_schedule.c b/platform/linux-generic/odp_schedule.c index 53670a71..8af27673 100644 --- a/platform/linux-generic/odp_schedule.c +++ b/platform/linux-generic/odp_schedule.c @@ -65,8 +65,11 @@ ODP_STATIC_ASSERT((ODP_SCHED_PRIO_NORMAL > 0) && /* Maximum number of pktio poll commands */ #define NUM_PKTIO_CMD (MAX_PKTIN * NUM_PKTIO) +/* Not a valid index */ +#define NULL_INDEX ((uint32_t)-1) + /* Not a valid poll command */ -#define PKTIO_CMD_INVALID ((uint32_t)-1) +#define PKTIO_CMD_INVALID NULL_INDEX /* Pktio command is free */ #define PKTIO_CMD_FREE PKTIO_CMD_INVALID @@ -90,7 +93,7 @@ ODP_STATIC_ASSERT((ODP_SCHED_PRIO_NORMAL > 0) && #define PRIO_QUEUE_MASK (PRIO_QUEUE_RING_SIZE - 1) /* Priority queue empty, not a valid queue index. */ -#define PRIO_QUEUE_EMPTY ((uint32_t)-1) +#define PRIO_QUEUE_EMPTY NULL_INDEX /* For best performance, the number of queues should be a power of two. */ ODP_STATIC_ASSERT(CHECK_IS_POWER2(ODP_CONFIG_QUEUES), @@ -127,7 +130,7 @@ ODP_STATIC_ASSERT((8 * sizeof(pri_mask_t)) >= QUEUES_PER_PRIO, /* Storage for stashed enqueue operation arguments */ typedef struct { odp_buffer_hdr_t *buf_hdr[QUEUE_MULTI_MAX]; - queue_entry_t *queue; + uint32_t queue_index; int num; } ordered_stash_t; @@ -152,7 +155,8 @@ typedef struct { odp_queue_t queue; odp_event_t ev_stash[MAX_DEQ]; struct { - queue_entry_t *src_queue; /**< Source queue entry */ + /* Source queue index */ + uint32_t src_queue; uint64_t ctx; /**< Ordered context id */ int stash_num; /**< Number of stashed enqueue operations */ uint8_t in_order; /**< Order status */ @@ -197,6 +201,19 @@ typedef struct { uint32_t cmd_index; } pktio_cmd_t; +/* Order context of a queue */ +typedef struct { + /* Current ordered context id */ + odp_atomic_u64_t ctx ODP_ALIGNED_CACHE; + + /* Next unallocated context id */ + odp_atomic_u64_t next_ctx; + + /* Array of ordered locks */ + odp_atomic_u64_t lock[CONFIG_QUEUE_MAX_ORD_LOCKS]; + +} order_context_t ODP_ALIGNED_CACHE; + typedef struct { pri_mask_t pri_mask[NUM_PRIO]; odp_spinlock_t mask_lock; @@ -230,6 +247,8 @@ typedef struct { int grp; int prio; int queue_per_prio; + int sync; + unsigned order_lock_count; } queue[ODP_CONFIG_QUEUES]; struct { @@ -237,6 +256,8 @@ typedef struct { int num_cmd; } pktio[NUM_PKTIO]; + order_context_t order[ODP_CONFIG_QUEUES]; + } sched_global_t; /* Global scheduler context */ @@ -259,6 +280,7 @@ static void sched_local_init(void) sched_local.thr = odp_thread_id(); sched_local.queue = ODP_QUEUE_INVALID; sched_local.queue_index = PRIO_QUEUE_EMPTY; + sched_local.ordered.src_queue = NULL_INDEX; id = sched_local.thr & (QUEUES_PER_PRIO - 1); @@ -488,16 +510,35 @@ static void pri_clr_queue(uint32_t queue_index, int prio) static int schedule_init_queue(uint32_t queue_index, const odp_schedule_param_t *sched_param) { + int i; int prio = sched_param->prio; pri_set_queue(queue_index, prio); sched->queue[queue_index].grp = sched_param->group; sched->queue[queue_index].prio = prio; sched->queue[queue_index].queue_per_prio = queue_per_prio(queue_index); + sched->queue[queue_index].sync = sched_param->sync; + sched->queue[queue_index].order_lock_count = sched_param->lock_count; + + odp_atomic_init_u64(&sched->order[queue_index].ctx, 0); + odp_atomic_init_u64(&sched->order[queue_index].next_ctx, 0); + + for (i = 0; i < CONFIG_QUEUE_MAX_ORD_LOCKS; i++) + odp_atomic_init_u64(&sched->order[queue_index].lock[i], 0); return 0; } +static inline int queue_is_atomic(uint32_t queue_index) +{ + return sched->queue[queue_index].sync == ODP_SCHED_SYNC_ATOMIC; +} + +static inline int queue_is_ordered(uint32_t queue_index) +{ + return sched->queue[queue_index].sync == ODP_SCHED_SYNC_ORDERED; +} + static void schedule_destroy_queue(uint32_t queue_index) { int prio = sched->queue[queue_index].prio; @@ -506,6 +547,11 @@ static void schedule_destroy_queue(uint32_t queue_index) sched->queue[queue_index].grp = 0; sched->queue[queue_index].prio = 0; sched->queue[queue_index].queue_per_prio = 0; + + if (queue_is_ordered(queue_index) && + odp_atomic_load_u64(&sched->order[queue_index].ctx) != + odp_atomic_load_u64(&sched->order[queue_index].next_ctx)) + ODP_ERR("queue reorder incomplete\n"); } static int poll_cmd_queue_idx(int pktio_index, int pktin_idx) @@ -606,20 +652,20 @@ static void schedule_release_atomic(void) } } -static inline int ordered_own_turn(queue_entry_t *queue) +static inline int ordered_own_turn(uint32_t queue_index) { uint64_t ctx; - ctx = odp_atomic_load_acq_u64(&queue->s.ordered.ctx); + ctx = odp_atomic_load_acq_u64(&sched->order[queue_index].ctx); return ctx == sched_local.ordered.ctx; } -static inline void wait_for_order(queue_entry_t *queue) +static inline void wait_for_order(uint32_t queue_index) { /* Busy loop to synchronize ordered processing */ while (1) { - if (ordered_own_turn(queue)) + if (ordered_own_turn(queue_index)) break; odp_cpu_pause(); } @@ -635,52 +681,54 @@ static inline void ordered_stash_release(void) int i; for (i = 0; i < sched_local.ordered.stash_num; i++) { - queue_entry_t *queue; + queue_entry_t *queue_entry; + uint32_t queue_index; odp_buffer_hdr_t **buf_hdr; int num; - queue = sched_local.ordered.stash[i].queue; + queue_index = sched_local.ordered.stash[i].queue_index; + queue_entry = get_qentry(queue_index); buf_hdr = sched_local.ordered.stash[i].buf_hdr; num = sched_local.ordered.stash[i].num; - queue_fn->enq_multi(qentry_to_int(queue), buf_hdr, num); + queue_fn->enq_multi(qentry_to_int(queue_entry), buf_hdr, num); } sched_local.ordered.stash_num = 0; } static inline void release_ordered(void) { + uint32_t qi; unsigned i; - queue_entry_t *queue; - queue = sched_local.ordered.src_queue; + qi = sched_local.ordered.src_queue; - wait_for_order(queue); + wait_for_order(qi); /* Release all ordered locks */ - for (i = 0; i < queue->s.param.sched.lock_count; i++) { + for (i = 0; i < sched->queue[qi].order_lock_count; i++) { if (!sched_local.ordered.lock_called.u8[i]) - odp_atomic_store_rel_u64(&queue->s.ordered.lock[i], + odp_atomic_store_rel_u64(&sched->order[qi].lock[i], sched_local.ordered.ctx + 1); } sched_local.ordered.lock_called.all = 0; - sched_local.ordered.src_queue = NULL; + sched_local.ordered.src_queue = NULL_INDEX; sched_local.ordered.in_order = 0; ordered_stash_release(); /* Next thread can continue processing */ - odp_atomic_add_rel_u64(&queue->s.ordered.ctx, 1); + odp_atomic_add_rel_u64(&sched->order[qi].ctx, 1); } static void schedule_release_ordered(void) { - queue_entry_t *queue; + uint32_t queue_index; - queue = sched_local.ordered.src_queue; + queue_index = sched_local.ordered.src_queue; - if (odp_unlikely(!queue || sched_local.num)) + if (odp_unlikely((queue_index == NULL_INDEX) || sched_local.num)) return; release_ordered(); @@ -688,7 +736,7 @@ static void schedule_release_ordered(void) static inline void schedule_release_context(void) { - if (sched_local.ordered.src_queue != NULL) + if (sched_local.ordered.src_queue != NULL_INDEX) release_ordered(); else schedule_release_atomic(); @@ -715,9 +763,9 @@ static int schedule_ord_enq_multi(queue_t q_int, void *buf_hdr[], int i; uint32_t stash_num = sched_local.ordered.stash_num; queue_entry_t *dst_queue = qentry_from_int(q_int); - queue_entry_t *src_queue = sched_local.ordered.src_queue; + uint32_t src_queue = sched_local.ordered.src_queue; - if (!sched_local.ordered.src_queue || sched_local.ordered.in_order) + if ((src_queue == NULL_INDEX) || sched_local.ordered.in_order) return 0; if (ordered_own_turn(src_queue)) { @@ -740,7 +788,7 @@ static int schedule_ord_enq_multi(queue_t q_int, void *buf_hdr[], return 0; } - sched_local.ordered.stash[stash_num].queue = dst_queue; + sched_local.ordered.stash[stash_num].queue_index = dst_queue->s.index; sched_local.ordered.stash[stash_num].num = num; for (i = 0; i < num; i++) sched_local.ordered.stash[stash_num].buf_hdr[i] = buf_hdr[i]; @@ -803,7 +851,7 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], prio > ODP_SCHED_PRIO_DEFAULT)) max_deq = MAX_DEQ / 2; - ordered = sched_cb_queue_is_ordered(qi); + ordered = queue_is_ordered(qi); /* Do not cache ordered events locally to improve * parallelism. Ordered context can only be released @@ -835,21 +883,18 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], if (ordered) { uint64_t ctx; - queue_entry_t *queue; odp_atomic_u64_t *next_ctx; - queue = get_qentry(qi); - next_ctx = &queue->s.ordered.next_ctx; - + next_ctx = &sched->order[qi].next_ctx; ctx = odp_atomic_fetch_inc_u64(next_ctx); sched_local.ordered.ctx = ctx; - sched_local.ordered.src_queue = queue; + sched_local.ordered.src_queue = qi; /* Continue scheduling ordered queues */ ring_enq(ring, PRIO_QUEUE_MASK, qi); - } else if (sched_cb_queue_is_atomic(qi)) { + } else if (queue_is_atomic(qi)) { /* Hold queue during atomic access */ sched_local.queue_index = qi; } else { @@ -1041,14 +1086,14 @@ static int schedule_multi(odp_queue_t *out_queue, uint64_t wait, static inline void order_lock(void) { - queue_entry_t *queue; + uint32_t queue_index; - queue = sched_local.ordered.src_queue; + queue_index = sched_local.ordered.src_queue; - if (!queue) + if (queue_index == NULL_INDEX) return; - wait_for_order(queue); + wait_for_order(queue_index); } static void order_unlock(void) @@ -1058,14 +1103,15 @@ static void order_unlock(void) static void schedule_order_lock(unsigned lock_index) { odp_atomic_u64_t *ord_lock; - queue_entry_t *queue; + uint32_t queue_index; - queue = sched_local.ordered.src_queue; + queue_index = sched_local.ordered.src_queue; - ODP_ASSERT(queue && lock_index <= queue->s.param.sched.lock_count && + ODP_ASSERT(queue_index != NULL_INDEX && + lock_index <= sched->queue[queue_index].order_lock_count && !sched_local.ordered.lock_called.u8[lock_index]); - ord_lock = &queue->s.ordered.lock[lock_index]; + ord_lock = &sched->order[queue_index].lock[lock_index]; /* Busy loop to synchronize ordered processing */ while (1) { @@ -1084,13 +1130,14 @@ static void schedule_order_lock(unsigned lock_index) static void schedule_order_unlock(unsigned lock_index) { odp_atomic_u64_t *ord_lock; - queue_entry_t *queue; + uint32_t queue_index; - queue = sched_local.ordered.src_queue; + queue_index = sched_local.ordered.src_queue; - ODP_ASSERT(queue && lock_index <= queue->s.param.sched.lock_count); + ODP_ASSERT(queue_index != NULL_INDEX && + lock_index <= sched->queue[queue_index].order_lock_count); - ord_lock = &queue->s.ordered.lock[lock_index]; + ord_lock = &sched->order[queue_index].lock[lock_index]; ODP_ASSERT(sched_local.ordered.ctx == odp_atomic_load_u64(ord_lock)); diff --git a/platform/linux-generic/odp_schedule_iquery.c b/platform/linux-generic/odp_schedule_iquery.c index 8d8dcc29..f315a4f0 100644 --- a/platform/linux-generic/odp_schedule_iquery.c +++ b/platform/linux-generic/odp_schedule_iquery.c @@ -71,6 +71,8 @@ typedef struct { /* Maximum number of pktio poll commands */ #define NUM_PKTIO_CMD (MAX_PKTIN * NUM_PKTIO) +/* Not a valid index */ +#define NULL_INDEX ((uint32_t)-1) /* Pktio command is free */ #define PKTIO_CMD_FREE ((uint32_t)-1) @@ -117,6 +119,19 @@ typedef struct { /* Forward declaration */ typedef struct sched_thread_local sched_thread_local_t; +/* Order context of a queue */ +typedef struct { + /* Current ordered context id */ + odp_atomic_u64_t ctx ODP_ALIGNED_CACHE; + + /* Next unallocated context id */ + odp_atomic_u64_t next_ctx; + + /* Array of ordered locks */ + odp_atomic_u64_t lock[CONFIG_QUEUE_MAX_ORD_LOCKS]; + +} order_context_t ODP_ALIGNED_CACHE; + typedef struct { odp_shm_t selfie; @@ -139,6 +154,8 @@ typedef struct { /* Quick reference to per thread context */ sched_thread_local_t *threads[ODP_THREAD_COUNT_MAX]; + + order_context_t order[ODP_CONFIG_QUEUES]; } sched_global_t; /* Per thread events cache */ @@ -154,7 +171,7 @@ typedef struct { /* Storage for stashed enqueue operation arguments */ typedef struct { odp_buffer_hdr_t *buf_hdr[QUEUE_MULTI_MAX]; - queue_entry_t *queue; + uint32_t queue_index; int num; } ordered_stash_t; @@ -195,7 +212,8 @@ struct sched_thread_local { sparse_bitmap_iterator_t iterators[NUM_SCHED_PRIO]; struct { - queue_entry_t *src_queue; /**< Source queue entry */ + /* Source queue index */ + uint32_t src_queue; uint64_t ctx; /**< Ordered context id */ int stash_num; /**< Number of stashed enqueue operations */ uint8_t in_order; /**< Order status */ @@ -314,6 +332,7 @@ static void sched_thread_local_reset(void) thread_local.thread = odp_thread_id(); thread_local.cache.queue = ODP_QUEUE_INVALID; + thread_local.ordered.src_queue = NULL_INDEX; odp_rwlock_init(&thread_local.lock); @@ -395,7 +414,7 @@ static int schedule_term_local(void) static int init_sched_queue(uint32_t queue_index, const odp_schedule_param_t *sched_param) { - int prio, group, thread; + int prio, group, thread, i; sched_prio_t *P; sched_group_t *G; sched_thread_local_t *local; @@ -428,6 +447,12 @@ static int init_sched_queue(uint32_t queue_index, memcpy(&sched->queues[queue_index], sched_param, sizeof(odp_schedule_param_t)); + odp_atomic_init_u64(&sched->order[queue_index].ctx, 0); + odp_atomic_init_u64(&sched->order[queue_index].next_ctx, 0); + + for (i = 0; i < CONFIG_QUEUE_MAX_ORD_LOCKS; i++) + odp_atomic_init_u64(&sched->order[queue_index].lock[i], 0); + /* Update all threads in this schedule group to * start check this queue index upon scheduling. */ @@ -502,6 +527,11 @@ static void destroy_sched_queue(uint32_t queue_index) __destroy_sched_queue(G, queue_index); odp_rwlock_write_unlock(&G->lock); + + if (sched->queues[queue_index].sync == ODP_SCHED_SYNC_ORDERED && + odp_atomic_load_u64(&sched->order[queue_index].ctx) != + odp_atomic_load_u64(&sched->order[queue_index].next_ctx)) + ODP_ERR("queue reorder incomplete\n"); } static int pktio_cmd_queue_hash(int pktio, int pktin) @@ -1070,20 +1100,20 @@ static void schedule_release_atomic(void) } } -static inline int ordered_own_turn(queue_entry_t *queue) +static inline int ordered_own_turn(uint32_t queue_index) { uint64_t ctx; - ctx = odp_atomic_load_acq_u64(&queue->s.ordered.ctx); + ctx = odp_atomic_load_acq_u64(&sched->order[queue_index].ctx); return ctx == thread_local.ordered.ctx; } -static inline void wait_for_order(queue_entry_t *queue) +static inline void wait_for_order(uint32_t queue_index) { /* Busy loop to synchronize ordered processing */ while (1) { - if (ordered_own_turn(queue)) + if (ordered_own_turn(queue_index)) break; odp_cpu_pause(); } @@ -1099,52 +1129,55 @@ static inline void ordered_stash_release(void) int i; for (i = 0; i < thread_local.ordered.stash_num; i++) { - queue_entry_t *queue; + queue_entry_t *queue_entry; + uint32_t queue_index; odp_buffer_hdr_t **buf_hdr; int num; - queue = thread_local.ordered.stash[i].queue; + queue_index = thread_local.ordered.stash[i].queue_index; + queue_entry = get_qentry(queue_index); buf_hdr = thread_local.ordered.stash[i].buf_hdr; num = thread_local.ordered.stash[i].num; - queue_fn->enq_multi(qentry_to_int(queue), buf_hdr, num); + queue_fn->enq_multi(qentry_to_int(queue_entry), buf_hdr, num); } thread_local.ordered.stash_num = 0; } static inline void release_ordered(void) { + uint32_t qi; unsigned i; - queue_entry_t *queue; - queue = thread_local.ordered.src_queue; + qi = thread_local.ordered.src_queue; - wait_for_order(queue); + wait_for_order(qi); /* Release all ordered locks */ - for (i = 0; i < queue->s.param.sched.lock_count; i++) { + for (i = 0; i < sched->queues[qi].lock_count; i++) { if (!thread_local.ordered.lock_called.u8[i]) - odp_atomic_store_rel_u64(&queue->s.ordered.lock[i], + odp_atomic_store_rel_u64(&sched->order[qi].lock[i], thread_local.ordered.ctx + 1); } thread_local.ordered.lock_called.all = 0; - thread_local.ordered.src_queue = NULL; + thread_local.ordered.src_queue = NULL_INDEX; thread_local.ordered.in_order = 0; ordered_stash_release(); /* Next thread can continue processing */ - odp_atomic_add_rel_u64(&queue->s.ordered.ctx, 1); + odp_atomic_add_rel_u64(&sched->order[qi].ctx, 1); } static void schedule_release_ordered(void) { - queue_entry_t *queue; + uint32_t queue_index; - queue = thread_local.ordered.src_queue; + queue_index = thread_local.ordered.src_queue; - if (odp_unlikely(!queue || thread_local.cache.count)) + if (odp_unlikely((queue_index == NULL_INDEX) || + thread_local.cache.count)) return; release_ordered(); @@ -1152,7 +1185,7 @@ static void schedule_release_ordered(void) static inline void schedule_release_context(void) { - if (thread_local.ordered.src_queue != NULL) + if (thread_local.ordered.src_queue != NULL_INDEX) release_ordered(); else schedule_release_atomic(); @@ -1164,9 +1197,9 @@ static int schedule_ord_enq_multi(queue_t q_int, void *buf_hdr[], int i; uint32_t stash_num = thread_local.ordered.stash_num; queue_entry_t *dst_queue = qentry_from_int(q_int); - queue_entry_t *src_queue = thread_local.ordered.src_queue; + uint32_t src_queue = thread_local.ordered.src_queue; - if (!thread_local.ordered.src_queue || thread_local.ordered.in_order) + if ((src_queue == NULL_INDEX) || thread_local.ordered.in_order) return 0; if (ordered_own_turn(src_queue)) { @@ -1189,7 +1222,7 @@ static int schedule_ord_enq_multi(queue_t q_int, void *buf_hdr[], return 0; } - thread_local.ordered.stash[stash_num].queue = dst_queue; + thread_local.ordered.stash[stash_num].queue_index = dst_queue->s.index; thread_local.ordered.stash[stash_num].num = num; for (i = 0; i < num; i++) thread_local.ordered.stash[stash_num].buf_hdr[i] = buf_hdr[i]; @@ -1202,14 +1235,14 @@ static int schedule_ord_enq_multi(queue_t q_int, void *buf_hdr[], static void order_lock(void) { - queue_entry_t *queue; + uint32_t queue_index; - queue = thread_local.ordered.src_queue; + queue_index = thread_local.ordered.src_queue; - if (!queue) + if (queue_index == NULL_INDEX) return; - wait_for_order(queue); + wait_for_order(queue_index); } static void order_unlock(void) @@ -1219,14 +1252,15 @@ static void order_unlock(void) static void schedule_order_lock(unsigned lock_index) { odp_atomic_u64_t *ord_lock; - queue_entry_t *queue; + uint32_t queue_index; - queue = thread_local.ordered.src_queue; + queue_index = thread_local.ordered.src_queue; - ODP_ASSERT(queue && lock_index <= queue->s.param.sched.lock_count && + ODP_ASSERT(queue_index != NULL_INDEX && + lock_index <= sched->queues[queue_index].lock_count && !thread_local.ordered.lock_called.u8[lock_index]); - ord_lock = &queue->s.ordered.lock[lock_index]; + ord_lock = &sched->order[queue_index].lock[lock_index]; /* Busy loop to synchronize ordered processing */ while (1) { @@ -1245,13 +1279,14 @@ static void schedule_order_lock(unsigned lock_index) static void schedule_order_unlock(unsigned lock_index) { odp_atomic_u64_t *ord_lock; - queue_entry_t *queue; + uint32_t queue_index; - queue = thread_local.ordered.src_queue; + queue_index = thread_local.ordered.src_queue; - ODP_ASSERT(queue && lock_index <= queue->s.param.sched.lock_count); + ODP_ASSERT(queue_index != NULL_INDEX && + lock_index <= sched->queues[queue_index].lock_count); - ord_lock = &queue->s.ordered.lock[lock_index]; + ord_lock = &sched->order[queue_index].lock[lock_index]; ODP_ASSERT(thread_local.ordered.ctx == odp_atomic_load_u64(ord_lock)); @@ -1275,7 +1310,7 @@ static inline bool is_ordered_queue(unsigned int queue_index) static void schedule_save_context(uint32_t queue_index, void *ptr) { - queue_entry_t *queue = ptr; + (void)ptr; if (is_atomic_queue(queue_index)) { thread_local.atomic = &sched->availables[queue_index]; @@ -1283,11 +1318,11 @@ static void schedule_save_context(uint32_t queue_index, void *ptr) uint64_t ctx; odp_atomic_u64_t *next_ctx; - next_ctx = &queue->s.ordered.next_ctx; + next_ctx = &sched->order[queue_index].next_ctx; ctx = odp_atomic_fetch_inc_u64(next_ctx); thread_local.ordered.ctx = ctx; - thread_local.ordered.src_queue = queue; + thread_local.ordered.src_queue = queue_index; } } commit 39e4ebd39f2d171bbfef5bfee6596033b83c56a0 Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Fri Jun 30 17:10:54 2017 +0300 linux-gen: sched: use config max ordered locks Use config file value for the number of ordered locks everywhere. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_config_internal.h b/platform/linux-generic/include/odp_config_internal.h index 3cff0045..469396df 100644 --- a/platform/linux-generic/include/odp_config_internal.h +++ b/platform/linux-generic/include/odp_config_internal.h @@ -27,7 +27,7 @@ /* * Maximum number of ordered locks per queue */ -#define CONFIG_QUEUE_MAX_ORD_LOCKS 4 +#define CONFIG_QUEUE_MAX_ORD_LOCKS 2 /* * Maximum number of packet IO resources diff --git a/platform/linux-generic/odp_schedule.c b/platform/linux-generic/odp_schedule.c index 69de7ac0..53670a71 100644 --- a/platform/linux-generic/odp_schedule.c +++ b/platform/linux-generic/odp_schedule.c @@ -121,12 +121,6 @@ ODP_STATIC_ASSERT((8 * sizeof(pri_mask_t)) >= QUEUES_PER_PRIO, /* Maximum number of dequeues */ #define MAX_DEQ CONFIG_BURST_SIZE -/* Maximum number of ordered locks per queue */ -#define MAX_ORDERED_LOCKS_PER_QUEUE 2 - -ODP_STATIC_ASSERT(MAX_ORDERED_LOCKS_PER_QUEUE <= CONFIG_QUEUE_MAX_ORD_LOCKS, - "Too_many_ordered_locks"); - /* Ordered stash size */ #define MAX_ORDERED_STASH 512 @@ -449,7 +443,7 @@ static inline int grp_update_tbl(void) static unsigned schedule_max_ordered_locks(void) { - return MAX_ORDERED_LOCKS_PER_QUEUE; + return CONFIG_QUEUE_MAX_ORD_LOCKS; } static inline int queue_per_prio(uint32_t queue_index) diff --git a/platform/linux-generic/odp_schedule_iquery.c b/platform/linux-generic/odp_schedule_iquery.c index 75f56e63..8d8dcc29 100644 --- a/platform/linux-generic/odp_schedule_iquery.c +++ b/platform/linux-generic/odp_schedule_iquery.c @@ -148,12 +148,6 @@ typedef struct { odp_event_t stash[MAX_DEQ], *top; } event_cache_t; -/* Maximum number of ordered locks per queue */ -#define MAX_ORDERED_LOCKS_PER_QUEUE 2 - -ODP_STATIC_ASSERT(MAX_ORDERED_LOCKS_PER_QUEUE <= CONFIG_QUEUE_MAX_ORD_LOCKS, - "Too_many_ordered_locks"); - /* Ordered stash size */ #define MAX_ORDERED_STASH 512 @@ -1266,7 +1260,7 @@ static void schedule_order_unlock(unsigned lock_index) static unsigned schedule_max_ordered_locks(void) { - return MAX_ORDERED_LOCKS_PER_QUEUE; + return CONFIG_QUEUE_MAX_ORD_LOCKS; } static inline bool is_atomic_queue(unsigned int queue_index) commit f9e7f458cf7151c75a9145272c62306b96f9ff27 Author: Petri Savolainen <petri.savolainen(a)linaro.org> Date: Fri Jun 30 17:10:53 2017 +0300 linux-gen: sched: remove schedule interface depedency to qentry Do not use queue internal type in schedule interface. Signed-off-by: Petri Savolainen <petri.savolainen(a)linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/platform/linux-generic/include/odp_schedule_if.h b/platform/linux-generic/include/odp_schedule_if.h index 5877a1cd..5abbb732 100644 --- a/platform/linux-generic/include/odp_schedule_if.h +++ b/platform/linux-generic/include/odp_schedule_if.h @@ -35,9 +35,10 @@ typedef int (*schedule_term_local_fn_t)(void); typedef void (*schedule_order_lock_fn_t)(void); typedef void (*schedule_order_unlock_fn_t)(void); typedef unsigned (*schedule_max_ordered_locks_fn_t)(void); -typedef void (*schedule_save_context_fn_t)(queue_entry_t *queue); +typedef void (*schedule_save_context_fn_t)(uint32_t queue_index, void *ptr); typedef struct schedule_fn_t { + int status_sync; schedule_pktio_start_fn_t pktio_start; schedule_thr_add_fn_t thr_add; schedule_thr_rem_fn_t thr_rem; @@ -45,7 +46,6 @@ typedef struct schedule_fn_t { schedule_init_queue_fn_t init_queue; schedule_destroy_queue_fn_t destroy_queue; schedule_sched_queue_fn_t sched_queue; - schedule_unsched_queue_fn_t unsched_queue; schedule_ord_enq_multi_fn_t ord_enq_multi; schedule_init_global_fn_t init_global; schedule_term_global_fn_t term_global; @@ -54,7 +54,11 @@ typedef struct schedule_fn_t { schedule_order_lock_fn_t order_lock; schedule_order_unlock_fn_t order_unlock; schedule_max_ordered_locks_fn_t max_ordered_locks; + + /* Called only when status_sync is set */ + schedule_unsched_queue_fn_t unsched_queue; schedule_save_context_fn_t save_context; + } schedule_fn_t; /* Interface towards the scheduler */ diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c index 19945584..2db95fc6 100644 --- a/platform/linux-generic/odp_queue.c +++ b/platform/linux-generic/odp_queue.c @@ -474,6 +474,7 @@ static inline int deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], int i, j; queue_entry_t *queue; int updated = 0; + int status_sync = sched_fn->status_sync; queue = qentry_from_int(q_int); LOCK(&queue->s.lock); @@ -490,7 +491,9 @@ static inline int deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], /* Already empty queue */ if (queue->s.status == QUEUE_STATUS_SCHED) { queue->s.status = QUEUE_STATUS_NOTSCHED; - sched_fn->unsched_queue(queue->s.index); + + if (status_sync) + sched_fn->unsched_queue(queue->s.index); } UNLOCK(&queue->s.lock); @@ -533,8 +536,8 @@ static inline int deq_multi(queue_t q_int, odp_buffer_hdr_t *buf_hdr[], if (hdr == NULL) queue->s.tail = NULL; - if (queue->s.type == ODP_QUEUE_TYPE_SCHED) - sched_fn->save_context(queue); + if (status_sync && queue->s.type == ODP_QUEUE_TYPE_SCHED) + sched_fn->save_context(queue->s.index, queue); UNLOCK(&queue->s.lock); diff --git a/platform/linux-generic/odp_schedule.c b/platform/linux-generic/odp_schedule.c index 814746c7..69de7ac0 100644 --- a/platform/linux-generic/odp_schedule.c +++ b/platform/linux-generic/odp_schedule.c @@ -22,9 +22,11 @@ #include <odp/api/sync.h> #include <odp/api/packet_io.h> #include <odp_ring_internal.h> -#include <odp_queue_internal.h> #include <odp_timer_internal.h> +/* Should remove this dependency */ +#include <odp_queue_internal.h> + /* Number of priority levels */ #define NUM_PRIO 8 @@ -1340,22 +1342,14 @@ static int schedule_sched_queue(uint32_t queue_index) return 0; } -static int schedule_unsched_queue(uint32_t queue_index ODP_UNUSED) -{ - return 0; -} - static int schedule_num_grps(void) { return NUM_SCHED_GRPS; } -static void schedule_save_context(queue_entry_t *queue ODP_UNUSED) -{ -} - /* Fill in scheduler interface */ const schedule_fn_t schedule_default_fn = { + .status_sync = 0, .pktio_start = schedule_pktio_start, .thr_add = schedule_thr_add, .thr_rem = schedule_thr_rem, @@ -1363,7 +1357,6 @@ const schedule_fn_t schedule_default_fn = { .init_queue = schedule_init_queue, .destroy_queue = schedule_destroy_queue, .sched_queue = schedule_sched_queue, - .unsched_queue = schedule_unsched_queue, .ord_enq_multi = schedule_ord_enq_multi, .init_global = schedule_init_global, .term_global = schedule_term_global, @@ -1372,7 +1365,8 @@ const schedule_fn_t schedule_default_fn = { .order_lock = order_lock, .order_unlock = order_unlock, .max_ordered_locks = schedule_max_ordered_locks, - .save_context = schedule_save_context + .unsched_queue = NULL, + .save_context = NULL }; /* Fill in scheduler API calls */ diff --git a/platform/linux-generic/odp_schedule_iquery.c b/platform/linux-generic/odp_schedule_iquery.c index b33ba7cd..75f56e63 100644 --- a/platform/linux-generic/odp_schedule_iquery.c +++ b/platform/linux-generic/odp_schedule_iquery.c @@ -12,7 +12,6 @@ #include <odp_internal.h> #include <odp_debug_internal.h> #include <odp_ring_internal.h> -#include <odp_queue_internal.h> #include <odp_buffer_internal.h> #include <odp_bitmap_internal.h> #include <odp/api/thread.h> @@ -25,6 +24,9 @@ #include <odp_config_internal.h> #include <odp_timer_internal.h> +/* Should remove this dependency */ +#include <odp_queue_internal.h> + /* Number of priority levels */ #define NUM_SCHED_PRIO 8 @@ -1267,11 +1269,23 @@ static unsigned schedule_max_ordered_locks(void) return MAX_ORDERED_LOCKS_PER_QUEUE; } -static void schedule_save_context(queue_entry_t *queue) +static inline bool is_atomic_queue(unsigned int queue_index) +{ + return (sched->queues[queue_index].sync == ODP_SCHED_SYNC_ATOMIC); +} + +static inline bool is_ordered_queue(unsigned int queue_index) +{ + return (sched->queues[queue_index].sync == ODP_SCHED_SYNC_ORDERED); +} + +static void schedule_save_context(uint32_t queue_index, void *ptr) { - if (queue->s.param.sched.sync == ODP_SCHED_SYNC_ATOMIC) { - thread_local.atomic = &sched->availables[queue->s.index]; - } else if (queue->s.param.sched.sync == ODP_SCHED_SYNC_ORDERED) { + queue_entry_t *queue = ptr; + + if (is_atomic_queue(queue_index)) { + thread_local.atomic = &sched->availables[queue_index]; + } else if (is_ordered_queue(queue_index)) { uint64_t ctx; odp_atomic_u64_t *next_ctx; @@ -1285,6 +1299,7 @@ static void schedule_save_context(queue_entry_t *queue) /* Fill in scheduler interface */ const schedule_fn_t schedule_iquery_fn = { + .status_sync = 1, .pktio_start = schedule_pktio_start, .thr_add = group_add_thread, .thr_rem = group_remove_thread, @@ -1292,7 +1307,6 @@ const schedule_fn_t schedule_iquery_fn = { .init_queue = init_sched_queue, .destroy_queue = destroy_sched_queue, .sched_queue = schedule_sched_queue, - .unsched_queue = schedule_unsched_queue, .ord_enq_multi = schedule_ord_enq_multi, .init_global = schedule_init_global, .term_global = schedule_term_global, @@ -1301,7 +1315,8 @@ const schedule_fn_t schedule_iquery_fn = { .order_lock = order_lock, .order_unlock = order_unlock, .max_ordered_locks = schedule_max_ordered_locks, - .save_context = schedule_save_context, + .unsched_queue = schedule_unsched_queue, + .save_context = schedule_save_context }; /* Fill in scheduler API calls */ @@ -1428,18 +1443,6 @@ static void thread_clear_interests(sched_thread_local_t *thread, } } -static inline bool is_atomic_queue(unsigned int queue_index) -{ - return (sched->queues[queue_index].sync - == ODP_SCHED_SYNC_ATOMIC); -} - -static inline bool is_ordered_queue(unsigned int queue_index) -{ - return (sched->queues[queue_index].sync - == ODP_SCHED_SYNC_ORDERED); -} - static inline bool compete_atomic_queue(unsigned int queue_index) { bool expected = sched->availables[queue_index]; diff --git a/platform/linux-generic/odp_schedule_sp.c b/platform/linux-generic/odp_schedule_sp.c index 14b7aa88..9829acc5 100644 --- a/platform/linux-generic/odp_schedule_sp.c +++ b/platform/linux-generic/odp_schedule_sp.c @@ -410,11 +410,6 @@ static int sched_queue(uint32_t qi) return 0; } -static int unsched_queue(uint32_t qi ODP_UNUSED) -{ - return 0; -} - static int ord_enq_multi(queue_t q_int, void *buf_hdr[], int num, int *ret) { @@ -830,12 +825,9 @@ static void order_unlock(void) { } -static void save_context(queue_entry_t *queue ODP_UNUSED) -{ -} - /* Fill in scheduler interface */ const schedule_fn_t schedule_sp_fn = { + .status_sync = 0, .pktio_start = pktio_start, .thr_add = thr_add, .thr_rem = thr_rem, @@ -843,16 +835,16 @@ const schedule_fn_t schedule_sp_fn = { .init_queue = init_queue, .destroy_queue = destroy_queue, .sched_queue = sched_queue, - .unsched_queue = unsched_queue, .ord_enq_multi = ord_enq_multi, .init_global = init_global, .term_global = term_global, .init_local = init_local, .term_local = term_local, - .order_lock = order_lock, - .order_unlock = order_unlock, + .order_lock = order_lock, + .order_unlock = order_unlock, .max_ordered_locks = max_ordered_locks, - .save_context = save_context + .unsched_queue = NULL, + .save_context = NULL }; /* Fill in scheduler API calls */ ----------------------------------------------------------------------- Summary of changes: .../linux-generic/include/odp_config_internal.h | 2 +- .../linux-generic/include/odp_queue_internal.h | 7 - platform/linux-generic/include/odp_schedule_if.h | 13 +- platform/linux-generic/odp_queue.c | 68 +-------- platform/linux-generic/odp_schedule.c | 161 +++++++++++++-------- platform/linux-generic/odp_schedule_iquery.c | 158 ++++++++++++-------- platform/linux-generic/odp_schedule_sp.c | 18 +-- 7 files changed, 211 insertions(+), 216 deletions(-) hooks/post-receive --

9 years, 1 month

[COMMIT] branch master updated. v1.15.0.0-31-g5a2ed3ee

by git＠git-us.linaro.org

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 5a2ed3eefb1a263139282c376741adb2c355bd5a (commit) from 2541e8688750f2a045aff3dbf544868be3ebf787 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 5a2ed3eefb1a263139282c376741adb2c355bd5a Author: Nikhil Agarwal <nikhil.agarwal(a)linaro.org> Date: Wed Jun 28 11:41:50 2017 +0530 test:queue:update order queue test for locks. Order queue test should reserve ordered locks only if ordered locks count is more than 1. Signed-off-by: Nikhil Agarwal <nikhil.agarwal(a)linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer(a)linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov(a)linaro.org> diff --git a/test/common_plat/validation/api/queue/queue.c b/test/common_plat/validation/api/queue/queue.c index 6a13c006..f08d811f 100644 --- a/test/common_plat/validation/api/queue/queue.c +++ b/test/common_plat/validation/api/queue/queue.c @@ -262,6 +262,7 @@ void queue_test_info(void) const char *const nq_order = "test_q_order"; odp_queue_info_t info; odp_queue_param_t param; + odp_queue_capability_t capability; char q_plain_ctx[] = "test_q_plain context data"; char q_order_ctx[] = "test_q_order context data"; unsigned lock_count; @@ -274,13 +275,18 @@ void queue_test_info(void) CU_ASSERT(odp_queue_context_set(q_plain, q_plain_ctx, sizeof(q_plain_ctx)) == 0); + memset(&capability, 0, sizeof(odp_queue_capability_t)); + CU_ASSERT(odp_queue_capability(&capability) == 0); /* Create a scheduled ordered queue with explicitly set params */ odp_queue_param_init(&param); param.type = ODP_QUEUE_TYPE_SCHED; param.sched.prio = ODP_SCHED_PRIO_NORMAL; param.sched.sync = ODP_SCHED_SYNC_ORDERED; param.sched.group = ODP_SCHED_GROUP_ALL; - param.sched.lock_count = 1; + if (capability.max_ordered_locks) + param.sched.lock_count = 1; + else + param.sched.lock_count = 0; param.context = q_order_ctx; q_order = odp_queue_create(nq_order, &param); CU_ASSERT(ODP_QUEUE_INVALID != q_order); ----------------------------------------------------------------------- Summary of changes: test/common_plat/validation/api/queue/queue.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) hooks/post-receive --

9 years, 1 month

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

git-lng-odp