[GitHub] [camel-quarkus] lburgazzoli opened a new issue #654: fhir native compilation is eating up lot of memory

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-quarkus] lburgazzoli opened a new issue #654: fhir native compilation is eating up lot of memory

GitBox
lburgazzoli opened a new issue #654: fhir native compilation is eating up lot of memory
URL: https://github.com/apache/camel-quarkus/issues/654
 
 
   The native compilation of the camel-fhir extension is quite demanding in term of resources and the native-process easily reach 14GB of RSS (which is the hard coded limit the native-image can reach) which make not possible to test the fhir extension as examplle with GitHub Actions.
   
   We should investigate how to optimize the build of the fhir extension
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-quarkus] lburgazzoli commented on issue #654: fhir native compilation is eating up lot of memory

GitBox
lburgazzoli commented on issue #654: fhir native compilation is eating up lot of memory
URL: https://github.com/apache/camel-quarkus/issues/654#issuecomment-577260688
 
 
   /cc @johnpoth

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-quarkus] lburgazzoli commented on issue #654: fhir native compilation is eating up lot of memory

GitBox
In reply to this post by GitBox
lburgazzoli commented on issue #654: fhir native compilation is eating up lot of memory
URL: https://github.com/apache/camel-quarkus/issues/654#issuecomment-577262389
 
 
   I did try to limit the memory the native-image can require but in that case either the build take too long or fails because of OutOfMemory

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-quarkus] johnpoth commented on issue #654: fhir native compilation is eating up lot of memory

GitBox
In reply to this post by GitBox
johnpoth commented on issue #654: fhir native compilation is eating up lot of memory
URL: https://github.com/apache/camel-quarkus/issues/654#issuecomment-577269000
 
 
   @lburgazzoli are you running native compilation with all FHIR specifications enabled ? (DSTU2...R5)
   
   Also what is the acceptable memory consumption when doing native compilation for a quarkus extension? Certainly not 14GB but I do feel like it's higher than average.
   
   When testing I noticed that the more Java reflection FHIR used, the more memory consumption occurred when doing native compilation.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-quarkus] lburgazzoli commented on issue #654: fhir native compilation is eating up lot of memory

GitBox
In reply to this post by GitBox
lburgazzoli commented on issue #654: fhir native compilation is eating up lot of memory
URL: https://github.com/apache/camel-quarkus/issues/654#issuecomment-577271596
 
 
   I'm running the test suite for the fhir extension so in native mode it should take into account only a single specification.
   
   I don't have any specific number in mind about how much memory a native-compilation can take but the public CI we can use have limitations around 7/8 GB so we should take that into account.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-quarkus] lburgazzoli edited a comment on issue #654: fhir native compilation is eating up lot of memory

GitBox
In reply to this post by GitBox
lburgazzoli edited a comment on issue #654: fhir native compilation is eating up lot of memory
URL: https://github.com/apache/camel-quarkus/issues/654#issuecomment-577271596
 
 
   I'm running the test suite for the fhir extension so in native mode it should take into account only a single specification.
   
   I don't have any specific number in mind about how much memory a native-compilation can take but the public CIs we can use have limitations around 7/8 GB so we should take that into account.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-quarkus] lburgazzoli commented on issue #654: fhir native compilation is eating up lot of memory

GitBox
In reply to this post by GitBox
lburgazzoli commented on issue #654: fhir native compilation is eating up lot of memory
URL: https://github.com/apache/camel-quarkus/issues/654#issuecomment-577276246
 
 
   I do not know if it could make sense to split the extension in code and specifications so we do not include any specification or 3th party library that is not strictly needed. I know that we do already have flags that tweak what we do include but the dependencies are still on the classpath and I'm not sure substrate can easily find what code paths are truly not possible thus they can be excluded from the analysis.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-quarkus] lburgazzoli edited a comment on issue #654: fhir native compilation is eating up lot of memory

GitBox
In reply to this post by GitBox
lburgazzoli edited a comment on issue #654: fhir native compilation is eating up lot of memory
URL: https://github.com/apache/camel-quarkus/issues/654#issuecomment-577276246
 
 
   I do not know if it could make sense to split the extension in core and specifications so we do not include any specification or 3th party library that is not strictly needed. I know that we do already have flags that tweak what we do include but the dependencies are still on the classpath and I'm not sure substrate can easily find what code paths are truly not possible thus they can be excluded from the analysis.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-quarkus] johnpoth commented on issue #654: fhir native compilation is eating up lot of memory

GitBox
In reply to this post by GitBox
johnpoth commented on issue #654: fhir native compilation is eating up lot of memory
URL: https://github.com/apache/camel-quarkus/issues/654#issuecomment-577279411
 
 
   @lburgazzoli I thought we were already doing that with the said flags. It would be cool if there was a substrate "flag" to ignore the specification libraries not used on the classpath. This would avoid creating as you suggested camel-fhir-dstu3 etc... quarkus extensions.
   
   Another options would be to add the specification Maven dependencies in Profiles.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-quarkus] lburgazzoli commented on issue #654: fhir native compilation is eating up lot of memory

GitBox
In reply to this post by GitBox
lburgazzoli commented on issue #654: fhir native compilation is eating up lot of memory
URL: https://github.com/apache/camel-quarkus/issues/654#issuecomment-577282183
 
 
   I do not even know if doing that would really lower the requirements, just the first thing that came into my mind :)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-quarkus] johnpoth commented on issue #654: fhir native compilation is eating up lot of memory

GitBox
In reply to this post by GitBox
johnpoth commented on issue #654: fhir native compilation is eating up lot of memory
URL: https://github.com/apache/camel-quarkus/issues/654#issuecomment-577283815
 
 
   ah ok :) I'll look into it

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-quarkus] johnpoth commented on issue #654: fhir native compilation is eating up lot of memory

GitBox
In reply to this post by GitBox
johnpoth commented on issue #654: fhir native compilation is eating up lot of memory
URL: https://github.com/apache/camel-quarkus/issues/654#issuecomment-577732548
 
 
   Quick update. I've tried removing the dependencies mentioned above and also removing the quarkus BuildSteps in [FhirProcessor](https://github.com/apache/camel-quarkus/blob/master/extensions/fhir/deployment/src/main/java/org/apache/camel/quarkus/component/fhir/deployment/FhirProcessor.java#L77) but I still get 11 GB of memory use. Further investigation is needed to understand whats taking up all this memory.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-quarkus] johnpoth commented on issue #654: fhir native compilation is eating up lot of memory

GitBox
In reply to this post by GitBox
johnpoth commented on issue #654: fhir native compilation is eating up lot of memory
URL: https://github.com/apache/camel-quarkus/issues/654#issuecomment-578209220
 
 
   The 11 GB mentioned above was mainly due to camel-quarkus-core-xml dependency. I've removed the dependency from camel-fhir.
   
   The main memory usage is now coming from enabling field reflection on the FHIR model classes [1]. I've sent a message to the quarkus guys to see if this is expected (the feedback so far is this is expected).
   
   During my testing i've also noticed camel-xslt extension requires > 12 GB which also might be a problem when testing through CIs.
   
   [1] https://github.com/apache/camel-quarkus/blob/master/extensions/fhir/deployment/src/main/java/org/apache/camel/quarkus/component/fhir/deployment/FhirProcessor.java#L134 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-quarkus] lburgazzoli commented on issue #654: fhir native compilation is eating up lot of memory

GitBox
In reply to this post by GitBox
lburgazzoli commented on issue #654: fhir native compilation is eating up lot of memory
URL: https://github.com/apache/camel-quarkus/issues/654#issuecomment-578213765
 
 
   You need to constraint the environment to see how much memory the native-image really requires as the native-image usually demand 80% of the physical memory with max 14 GB so it may be ok to see other extension such has the xslt one to consume 12 GB.
   
   So if by removing xml you are now down to 11 GB there may be a change it will work with a constrained environment (you can set the native image max memory to something like 7 GB to validate)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services