how to store values into some persistent object when using File Endpoints

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

how to store values into some persistent object when using File Endpoints

Thomas Humphries
I have checked the docs, forums and FAQs but I can't find a way to do what
I need.

We have SpringBoot+Camel RESTful service:
Camel Core 2.22.1, Spring Boot starter 2.22.1 under Spring Boot 2.0.3.

We have the following (slightly obfuscated for IP reasons) rest routes to
receive, process and return a file:

public void configure() throws Exception {
.unmarshal().json(JsonLibrary.Jackson, Options.class)

public void configure() throws Exception {
.setHeader(Exchange.FILE_NAME, () -> UUID.randomUUID().toString())  // use
a UUID for inprocess filename
   .setBody((ex) -> buildOptions(ex))   // appends  json as function.options
   .marshal().json(JsonLibrary.Jackson, true)
   .log(LoggingLevel.INFO, "function.options: ${in.body}")
   .setBody((ex) -> new ProcessResponse((String)
   .marshal().json(JsonLibrary.Jackson, true);

.when((ex) -> Paths.get("function/output", (String)
.setHeader(Exchange.HTTP_RESPONSE_CODE, constant("202"));    /// send 202
until file complete

.log("returning file content")
.setBody((ex) -> Paths.get("func/output", (String)
.setHeader(Exchange.CONTENT_TYPE, constant(MediaType.TEXT_PLAIN_VALUE));

The *OptionsToHeadersProcessor* puts a filename and the Config options into

public class OptionsToHeadersProcessor implements Processor{

public static final String PROCESS_PROFILE = "PROCESS_PROFILE";
public static final String FILE_NAME = "FILE_NAME";

public void process(Exchange exchange) throws Exception {
Options options = exchange.getIn().getBody(Options.class);
exchange.getIn().setHeader(FILE_NAME, options.getInputFile());
exchange.getIn().setHeader(PROCESS_PROFILE, options.getConfiguration());
 //as header


The processing is working fine, files are returning etc.  We have log4j
performing log appending (rollingfile) for the service, also OK.

However, we now want to enhance by generating a statistical set of data
about each file processed.
Statistics required include: -

   -  process.start time, end time and duration,
   -  one or more method timings - also start, end dutration (from within
   the main *functionEngineProcessor*)
   -  data about the composition of the input file (e.g. bytes/lines) in
   total, and for specific sections within the file.

We also need to compile all those stats into a *single (json) text* - a
single line of json - before we emit to a file.

This last requirement is causing the most issue; I think you can see that
if I need to put, say start, end time and duration together to log them
(even using log4j), I need to persist the start time value across the
processors, so that my statistics processor can calculate the final end
time and duration.

I have tried using an intercept, just before the first endpoint definition
in  *FileReceiverRouteBuilder*.java:

.log(LoggingLevel.INFO, "intercepted send to output for metrics")

However *metricsProcessor* cannot retrieve anything from headers (not even
PROCESS_PROFILE or FILE_NAME) since apparently file endpints do not persist
properties or headers.

The "token" is retrievable, but that is tied to the client REST signature
and I cant extend that without a lot of disruption.  I can derive a
filename from the token, so processor is able to read the input and output
files and gather some of the required metrics.  I would rather use/derive
from FILE_NAME, if it were available.

I therefore am unsure how to store values into some persistent object for
retreival / use in the metrics processor, since passing properties and
headers between routes isn't available with File EndPoints.

My current reading seems to imply JMS is a way to achieve this? I'm new to
camel etc and I wanted to see if I'm missing something obvious.

We would rather not propose any downstream concatenation if possible.

Any help much appreciated!


*Thomas Humphries*