Hi toolchain champions,
[please keep me in cc as I'm not subscribed to linaro-toolchain@lists.linaro.org]
In OP-TEE we are going to activate a pager which is an integrated part of the statically linked secure OS binary (compiled for ARMv7/Aarch32 now, but at some point also Aarch64).
The pager in OP-TEE allows use of more memory than the amount of available physical memory. This makes it possible to for instance have an OP-TEE binary that requires more memory than the amount of available memory. What the pager does is to map a physical page at the virtual address where the memory is needed and populate it which what is expected on demand. The pager also unmaps physical pages that hasn't been used in a while to be able to recycle it.
The code used by the pager to map and populate a page must always be mapped since we would otherwise get a deadlock. The problem is that the pager is also part of OP-TEE so we need to partition the binary in a way that all code needed to handle a page fault is in one area in the binary and always mapped.
Annotating functions and such as it's done in the Linux kernel with __init will not scale here since the pager will need routines from "third-party" libraries. We can make small changes to the libraries but identifying and annotating everything needed by the pager is too much. We would also run into troubles with strings.
I have a couple ideas below that I need help exploring.
What if we do an incremental linking of the entire TEE Core with garbage collect only keeping the entry functions of the pager? Then we would get an object file with everything the pager depends on included but not much more. It would be easy to put this single object file in a separate part of the OP-TEE binary. The procedure would be something like:
Compile everything with -ffunction-sections -fdata-sections ld -i --gc-sections -u pager_entry -o pager_code.o $(objs) $(link-ldadd) $(libgcc) ld $(link-ldflags) pager_code.o $(objs) $(link-ldadd) $(libgcc)
But the problem comes with linking in the rest of the code in the last step, we would get lots of multiple defined symbols. We could create a libtee_core.a file where we put all the $(objs) files and the linker would only use the needed object files. But we would still have some multiple defined symbols left since each .o file contains more than just one section.
Any ideas how to solve this?
We could perhaps split each .o file into several .o files each with only one section. Would it work? Would it make the resulting binary larger or inefficient?
Another option could be to mark all symbols in libtee_core.a and other libaries as weak, but the problem here is that we already have some weak functions in TEE Core so this would break that. Perhaps if it would be possible to have different levels of weakness.
Any ideas are welcome, either along this path or different approaches.
Regards, Jens
On 30 October 2014 11:20, Jens Wiklander jens.wiklander@linaro.org wrote:
Hi toolchain champions,
[please keep me in cc as I'm not subscribed to linaro-toolchain@lists.linaro.org]
In OP-TEE we are going to activate a pager which is an integrated part of the statically linked secure OS binary (compiled for ARMv7/Aarch32 now, but at some point also Aarch64).
The pager in OP-TEE allows use of more memory than the amount of available physical memory. This makes it possible to for instance have an OP-TEE binary that requires more memory than the amount of available memory. What the pager does is to map a physical page at the virtual address where the memory is needed and populate it which what is expected on demand. The pager also unmaps physical pages that hasn't been used in a while to be able to recycle it.
The code used by the pager to map and populate a page must always be mapped since we would otherwise get a deadlock. The problem is that the pager is also part of OP-TEE so we need to partition the binary in a way that all code needed to handle a page fault is in one area in the binary and always mapped.
Annotating functions and such as it's done in the Linux kernel with __init will not scale here since the pager will need routines from "third-party" libraries. We can make small changes to the libraries but identifying and annotating everything needed by the pager is too much. We would also run into troubles with strings.
I have a couple ideas below that I need help exploring.
What if we do an incremental linking of the entire TEE Core with garbage collect only keeping the entry functions of the pager? Then we would get an object file with everything the pager depends on included but not much more. It would be easy to put this single object file in a separate part of the OP-TEE binary. The procedure would be something like:
Compile everything with -ffunction-sections -fdata-sections ld -i --gc-sections -u pager_entry -o pager_code.o $(objs) $(link-ldadd) $(libgcc) ld $(link-ldflags) pager_code.o $(objs) $(link-ldadd) $(libgcc)
But the problem comes with linking in the rest of the code in the last step, we would get lots of multiple defined symbols. We could create a libtee_core.a file where we put all the $(objs) files and the linker would only use the needed object files. But we would still have some multiple defined symbols left since each .o file contains more than just one section.
Any ideas how to solve this?
If you can confirm that the first step correctly produces a kernel with all the pager dependencies, couldn't you wrap all remaining code into a static library and link against that? This would solve the multiple definition issue as the archive members will be pulled in on demand. The only thing you need to ensure is that the granularity is sufficient, i.e., if the linker decides to pull a .o from the static lib, it will pull all contents including symbols that may conflict, but I would expect the -ffunction-sections -fdata-sections to take care of that.
Another trick that I know bionic uses for the loader is to prefix all symbols using objcopy --prefix-symbols=... That way, you can work around the multiple definitions, but you may end up with duplicate data which is a bit of a hazard.
+Etienne +Jean-Michel
On 30 October 2014 11:31, Ard Biesheuvel ard.biesheuvel@linaro.org wrote:
On 30 October 2014 11:20, Jens Wiklander jens.wiklander@linaro.org wrote:
Hi toolchain champions,
[please keep me in cc as I'm not subscribed to linaro-toolchain@lists.linaro.org]
In OP-TEE we are going to activate a pager which is an integrated part of the statically linked secure OS binary (compiled for ARMv7/Aarch32 now,
but
at some point also Aarch64).
The pager in OP-TEE allows use of more memory than the amount of
available
physical memory. This makes it possible to for instance have an OP-TEE binary that requires more memory than the amount of available memory.
What
the pager does is to map a physical page at the virtual address where the memory is needed and populate it which what is expected on demand. The
pager
also unmaps physical pages that hasn't been used in a while to be able to recycle it.
The code used by the pager to map and populate a page must always be
mapped
since we would otherwise get a deadlock. The problem is that the pager is also part of OP-TEE so we need to partition the binary in a way that all code needed to handle a page fault is in one area in the binary and
always
mapped.
Annotating functions and such as it's done in the Linux kernel with
__init
will not scale here since the pager will need routines from "third-party" libraries. We can make small changes to the libraries but identifying and annotating everything needed by the pager is too much. We would also run into troubles with strings.
I have a couple ideas below that I need help exploring.
What if we do an incremental linking of the entire TEE Core with garbage collect only keeping the entry functions of the pager? Then we would get
an
object file with everything the pager depends on included but not much
more.
It would be easy to put this single object file in a separate part of the OP-TEE binary. The procedure would be something like:
Compile everything with -ffunction-sections -fdata-sections ld -i --gc-sections -u pager_entry -o pager_code.o $(objs) $(link-ldadd) $(libgcc) ld $(link-ldflags) pager_code.o $(objs) $(link-ldadd) $(libgcc)
But the problem comes with linking in the rest of the code in the last
step,
we would get lots of multiple defined symbols. We could create a libtee_core.a file where we put all the $(objs) files and the linker
would
only use the needed object files. But we would still have some multiple defined symbols left since each .o file contains more than just one
section.
Any ideas how to solve this?
If you can confirm that the first step correctly produces a kernel with all the pager dependencies, couldn't you wrap all remaining code into a static library and link against that? This would solve the multiple definition issue as the archive members will be pulled in on demand. The only thing you need to ensure is that the granularity is sufficient, i.e., if the linker decides to pull a .o from the static lib, it will pull all contents including symbols that may conflict, but I would expect the -ffunction-sections -fdata-sections to take care of that.
Another trick that I know bionic uses for the loader is to prefix all symbols using objcopy --prefix-symbols=... That way, you can work around the multiple definitions, but you may end up with duplicate data which is a bit of a hazard.
-- Ard.
We could perhaps split each .o file into several .o files each with only
one
section. Would it work? Would it make the resulting binary larger or inefficient?
Another option could be to mark all symbols in libtee_core.a and other libaries as weak, but the problem here is that we already have some weak functions in TEE Core so this would break that. Perhaps if it would be possible to have different levels of weakness.
Any ideas are welcome, either along this path or different approaches.
Regards, Jens
On 30 October 2014 11:31, Ard Biesheuvel ard.biesheuvel@linaro.org wrote:
On 30 October 2014 11:20, Jens Wiklander jens.wiklander@linaro.org wrote:
Hi toolchain champions,
[please keep me in cc as I'm not subscribed to linaro-toolchain@lists.linaro.org]
In OP-TEE we are going to activate a pager which is an integrated part of the statically linked secure OS binary (compiled for ARMv7/Aarch32 now, but at some point also Aarch64).
The pager in OP-TEE allows use of more memory than the amount of available physical memory. This makes it possible to for instance have an OP-TEE binary that requires more memory than the amount of available memory. What the pager does is to map a physical page at the virtual address where the memory is needed and populate it which what is expected on demand. The pager also unmaps physical pages that hasn't been used in a while to be able to recycle it.
The code used by the pager to map and populate a page must always be mapped since we would otherwise get a deadlock. The problem is that the pager is also part of OP-TEE so we need to partition the binary in a way that all code needed to handle a page fault is in one area in the binary and always mapped.
Annotating functions and such as it's done in the Linux kernel with __init will not scale here since the pager will need routines from "third-party" libraries. We can make small changes to the libraries but identifying and annotating everything needed by the pager is too much. We would also run into troubles with strings.
I have a couple ideas below that I need help exploring.
What if we do an incremental linking of the entire TEE Core with garbage collect only keeping the entry functions of the pager? Then we would get an object file with everything the pager depends on included but not much more. It would be easy to put this single object file in a separate part of the OP-TEE binary. The procedure would be something like:
Compile everything with -ffunction-sections -fdata-sections ld -i --gc-sections -u pager_entry -o pager_code.o $(objs) $(link-ldadd) $(libgcc) ld $(link-ldflags) pager_code.o $(objs) $(link-ldadd) $(libgcc)
But the problem comes with linking in the rest of the code in the last step, we would get lots of multiple defined symbols. We could create a libtee_core.a file where we put all the $(objs) files and the linker would only use the needed object files. But we would still have some multiple defined symbols left since each .o file contains more than just one section.
Any ideas how to solve this?
If you can confirm that the first step correctly produces a kernel with all the pager dependencies, couldn't you wrap all remaining code into a static library and link against that? This would solve the multiple definition issue as the archive members will be pulled in on demand. The only thing you need to ensure is that the granularity is sufficient, i.e., if the linker decides to pull a .o from the static lib, it will pull all contents including symbols that may conflict, but I would expect the -ffunction-sections -fdata-sections to take care of that.
Hmm, I suppose that is what you meant with libtee_core.a?
I suppose you can use objcopy to separate .text and .data sections into separate .o files and wrap those up into an archive. There is also --localize-symbol and --weaken-symbol options in objcopy, but I making all symbols weak in libtee_core.a is probably not the way to go.
Is there any way to make sure the conflicts are over .text or .rodata symbols only? That would at least eliminate the duplicate data hazard, leaving only the code bloat due to duplication.
On Thu, Oct 30, 2014 at 12:22 PM, Ard Biesheuvel ard.biesheuvel@linaro.org wrote:
On 30 October 2014 11:31, Ard Biesheuvel ard.biesheuvel@linaro.org wrote:
On 30 October 2014 11:20, Jens Wiklander jens.wiklander@linaro.org
wrote:
Hi toolchain champions,
[please keep me in cc as I'm not subscribed to linaro-toolchain@lists.linaro.org]
In OP-TEE we are going to activate a pager which is an integrated part
of
the statically linked secure OS binary (compiled for ARMv7/Aarch32 now,
but
at some point also Aarch64).
The pager in OP-TEE allows use of more memory than the amount of
available
physical memory. This makes it possible to for instance have an OP-TEE binary that requires more memory than the amount of available memory.
What
the pager does is to map a physical page at the virtual address where
the
memory is needed and populate it which what is expected on demand. The
pager
also unmaps physical pages that hasn't been used in a while to be able
to
recycle it.
The code used by the pager to map and populate a page must always be
mapped
since we would otherwise get a deadlock. The problem is that the pager
is
also part of OP-TEE so we need to partition the binary in a way that all code needed to handle a page fault is in one area in the binary and
always
mapped.
Annotating functions and such as it's done in the Linux kernel with
__init
will not scale here since the pager will need routines from
"third-party"
libraries. We can make small changes to the libraries but identifying
and
annotating everything needed by the pager is too much. We would also run into troubles with strings.
I have a couple ideas below that I need help exploring.
What if we do an incremental linking of the entire TEE Core with garbage collect only keeping the entry functions of the pager? Then we would
get an
object file with everything the pager depends on included but not much
more.
It would be easy to put this single object file in a separate part of
the
OP-TEE binary. The procedure would be something like:
Compile everything with -ffunction-sections -fdata-sections ld -i --gc-sections -u pager_entry -o pager_code.o $(objs) $(link-ldadd) $(libgcc) ld $(link-ldflags) pager_code.o $(objs) $(link-ldadd) $(libgcc)
But the problem comes with linking in the rest of the code in the last
step,
we would get lots of multiple defined symbols. We could create a libtee_core.a file where we put all the $(objs) files and the linker
would
only use the needed object files. But we would still have some multiple defined symbols left since each .o file contains more than just one
section.
Any ideas how to solve this?
If you can confirm that the first step correctly produces a kernel with all the pager dependencies, couldn't you wrap all remaining code into a static library and link against that? This would solve the multiple definition issue as the archive members will be pulled in on demand. The only thing you need to ensure is that the granularity is sufficient, i.e., if the linker decides to pull a .o from the static lib, it will pull all contents including symbols that may conflict, but I would expect the -ffunction-sections -fdata-sections to take care of that.
Hmm, I suppose that is what you meant with libtee_core.a?
Yes. :-)
I suppose you can use objcopy to separate .text and .data sections into separate .o files and wrap those up into an archive. There is also --localize-symbol and --weaken-symbol options in objcopy, but I making all symbols weak in libtee_core.a is probably not the way to go.
Is there any way to make sure the conflicts are over .text or .rodata symbols only? That would at least eliminate the duplicate data hazard, leaving only the code bloat due to duplication.
Yes, we could let all .data and .bss go into the pager, but I don't like the code bloat.
Regards, Jens
-- Ard.
Another trick that I know bionic uses for the loader is to prefix all symbols using objcopy --prefix-symbols=... That way, you can work around the multiple definitions, but you may end up with duplicate data which is a bit of a hazard.
-- Ard.
We could perhaps split each .o file into several .o files each with
only one
section. Would it work? Would it make the resulting binary larger or inefficient?
Another option could be to mark all symbols in libtee_core.a and other libaries as weak, but the problem here is that we already have some weak functions in TEE Core so this would break that. Perhaps if it would be possible to have different levels of weakness.
Any ideas are welcome, either along this path or different approaches.
Regards, Jens
On Thu, Oct 30, 2014 at 1:18 PM, Jens Wiklander jens.wiklander@linaro.org wrote:
On Thu, Oct 30, 2014 at 12:22 PM, Ard Biesheuvel < ard.biesheuvel@linaro.org> wrote:
On 30 October 2014 11:31, Ard Biesheuvel ard.biesheuvel@linaro.org wrote:
On 30 October 2014 11:20, Jens Wiklander jens.wiklander@linaro.org
wrote:
Hi toolchain champions,
[please keep me in cc as I'm not subscribed to linaro-toolchain@lists.linaro.org]
In OP-TEE we are going to activate a pager which is an integrated part
of
the statically linked secure OS binary (compiled for ARMv7/Aarch32
now, but
at some point also Aarch64).
The pager in OP-TEE allows use of more memory than the amount of
available
physical memory. This makes it possible to for instance have an OP-TEE binary that requires more memory than the amount of available memory.
What
the pager does is to map a physical page at the virtual address where
the
memory is needed and populate it which what is expected on demand. The
pager
also unmaps physical pages that hasn't been used in a while to be able
to
recycle it.
The code used by the pager to map and populate a page must always be
mapped
since we would otherwise get a deadlock. The problem is that the pager
is
also part of OP-TEE so we need to partition the binary in a way that
all
code needed to handle a page fault is in one area in the binary and
always
mapped.
Annotating functions and such as it's done in the Linux kernel with
__init
will not scale here since the pager will need routines from
"third-party"
libraries. We can make small changes to the libraries but identifying
and
annotating everything needed by the pager is too much. We would also
run
into troubles with strings.
I have a couple ideas below that I need help exploring.
What if we do an incremental linking of the entire TEE Core with
garbage
collect only keeping the entry functions of the pager? Then we would
get an
object file with everything the pager depends on included but not much
more.
It would be easy to put this single object file in a separate part of
the
OP-TEE binary. The procedure would be something like:
Compile everything with -ffunction-sections -fdata-sections ld -i --gc-sections -u pager_entry -o pager_code.o $(objs)
$(link-ldadd)
$(libgcc) ld $(link-ldflags) pager_code.o $(objs) $(link-ldadd) $(libgcc)
But the problem comes with linking in the rest of the code in the last
step,
we would get lots of multiple defined symbols. We could create a libtee_core.a file where we put all the $(objs) files and the linker
would
only use the needed object files. But we would still have some multiple defined symbols left since each .o file contains more than just one
section.
Any ideas how to solve this?
After lots of experimentation I've finally found something that seems to work. What I do is something like: Compile everything with -ffunction-sections -fdata-sections ld -i --gc-sections -u pager_entry -o pager_code.o $(objs) $(link-ldadd) $(libgcc) to get all required sections. Then I have a script generating a list of all the sections in a format suitable to include in our linker script. After that I link the OP-TEE binary as usual and voila, all pager related sections are in one area.
There's one small problem though, it seems that even if I compile everything with "-ffunction-sections -fdata-sections" there's still some stuff emitted in .rodata. It seems that for instance: printf("Hello world"); would result in putting the string "Hello world" in .rodata. Any ideas how to get these constant strings in one .rodata.something each instead?
Regards, Jens
linaro-toolchain@lists.linaro.org
-
Ard Biesheuvel -
Jens Wiklander -
Pascal Brand