[GitHub] [camel] Nayananga opened a new pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

classic Classic list List threaded Threaded
53 messages Options
123
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel] Nayananga opened a new pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

GitBox

Nayananga opened a new pull request #3897:
URL: https://github.com/apache/camel/pull/3897


   This is a draft pull request
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel] oscerd commented on a change in pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

GitBox

oscerd commented on a change in pull request #3897:
URL: https://github.com/apache/camel/pull/3897#discussion_r436470452



##########
File path: components/camel-minio/ReadMe.txt
##########
@@ -0,0 +1,33 @@
+
+<!--
+
+    Licensed to the Apache Software Foundation (ASF) under one or more
+    contributor license agreements.  See the NOTICE file distributed with
+    this work for additional information regarding copyright ownership.
+    The ASF licenses this file to You under the Apache License, Version 2.0
+    (the "License"); you may not use this file except in compliance with
+    the License.  You may obtain a copy of the License at
+
+         http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+
+-->
+
+Camel Component Project
+=======================
+
+This project is a template of a Camel component.
+
+To build this project use
+
+    mvn install
+
+For more help see the Apache Camel documentation:
+
+    http://camel.apache.org/writing-components.html
+    

Review comment:
       This is not needed. You could remove this file.

##########
File path: components/camel-minio/pom.xml
##########
@@ -0,0 +1,140 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+    Licensed to the Apache Software Foundation (ASF) under one or more
+    contributor license agreements.  See the NOTICE file distributed with
+    this work for additional information regarding copyright ownership.
+    The ASF licenses this file to You under the Apache License, Version 2.0
+    (the "License"); you may not use this file except in compliance with
+    the License.  You may obtain a copy of the License at
+
+         http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+
+  <modelVersion>4.0.0</modelVersion>
+
+  <parent>
+    <artifactId>components</artifactId>
+    <groupId>org.apache.camel</groupId>
+    <version>3.4.0-SNAPSHOT</version>
+  </parent>
+
+  <artifactId>camel-minio</artifactId>
+  <packaging>jar</packaging>
+
+  <name>Camel :: MinIO</name>
+  <description>A Camel MinIO cloud storage server Component </description>
+
+  <properties>
+    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
+  </properties>
+
+  <dependencyManagement>
+    <dependencies>
+      <!-- Camel BOM -->
+      <dependency>
+        <groupId>org.apache.camel</groupId>

Review comment:
       No need for the camel-bom here. Take as example the other components in the components folder.

##########
File path: components/pom.xml
##########
@@ -382,6 +382,7 @@
         <module>camel-zipkin</module>
         <module>camel-zookeeper-master</module>
         <module>camel-zookeeper</module>
+    <module>camel-minio</module>

Review comment:
       Please add the component in the correct order.

##########
File path: components/camel-minio/src/main/java/org/apache/camel/EventBusHelper.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.camel;
+
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.Consumer;
+
+/**
+ * Used for demonstrations purpose to simulate some external system event bus/broker, where messages are sent to, and
+ * this component can consume from.
+ */
+public class EventBusHelper {

Review comment:
       This class can be removed.

##########
File path: components/camel-minio/src/main/java/org/apache/camel/MinIOConsumer.java
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.camel;
+
+import org.apache.camel.Exchange;
+import org.apache.camel.Processor;
+import org.apache.camel.RuntimeCamelException;
+import org.apache.camel.support.DefaultConsumer;
+
+import java.util.concurrent.ExecutorService;
+
+public class MinIOConsumer extends DefaultConsumer {
+    private final MinIOEndpoint endpoint;
+    private final EventBusHelper eventBusHelper;
+
+    private ExecutorService executorService;
+
+    public MinIOConsumer(MinIOEndpoint endpoint, Processor processor) {
+        super(endpoint, processor);
+        this.endpoint = endpoint;
+        eventBusHelper = EventBusHelper.getInstance();

Review comment:
       Remove the eventBusHelper, except it is something really useful to the basic feature of component, but I don't think so.

##########
File path: components/camel-minio/src/main/java/org/apache/camel/MinIOComponent.java
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.camel;
+
+import java.util.Map;
+
+import org.apache.camel.CamelContext;
+import org.apache.camel.Endpoint;
+
+import org.apache.camel.support.DefaultComponent;
+
+@org.apache.camel.spi.annotations.Component("MinIO")

Review comment:
       You can use directly Component annotation without the whole package.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel] Nayananga commented on a change in pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

GitBox
In reply to this post by GitBox

Nayananga commented on a change in pull request #3897:
URL: https://github.com/apache/camel/pull/3897#discussion_r437672436



##########
File path: components/camel-minio/src/main/java/org/apache/camel/EventBusHelper.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.camel;
+
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.Consumer;
+
+/**
+ * Used for demonstrations purpose to simulate some external system event bus/broker, where messages are sent to, and
+ * this component can consume from.
+ */
+public class EventBusHelper {

Review comment:
       Hi thank you for the suggestion, and I'll remove this for sure. But I'll keep it for a while for some random hello world kind tests while on the go. :)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel] Nayananga commented on a change in pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

GitBox
In reply to this post by GitBox

Nayananga commented on a change in pull request #3897:
URL: https://github.com/apache/camel/pull/3897#discussion_r437673851



##########
File path: components/camel-minio/src/main/java/org/apache/camel/MinIOConsumer.java
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.camel;
+
+import org.apache.camel.Exchange;
+import org.apache.camel.Processor;
+import org.apache.camel.RuntimeCamelException;
+import org.apache.camel.support.DefaultConsumer;
+
+import java.util.concurrent.ExecutorService;
+
+public class MinIOConsumer extends DefaultConsumer {
+    private final MinIOEndpoint endpoint;
+    private final EventBusHelper eventBusHelper;
+
+    private ExecutorService executorService;
+
+    public MinIOConsumer(MinIOEndpoint endpoint, Processor processor) {
+        super(endpoint, processor);
+        this.endpoint = endpoint;
+        eventBusHelper = EventBusHelper.getInstance();

Review comment:
       Hi thank you for the suggestion, and I'll remove this for sure. But I'll keep it for a while for some random hello world kind tests while on the go. :)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel] DenisIstomin commented on a change in pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

GitBox
In reply to this post by GitBox

DenisIstomin commented on a change in pull request #3897:
URL: https://github.com/apache/camel/pull/3897#discussion_r439466333



##########
File path: components/camel-minio/src/main/java/org/apache/camel/MinIOEndpoint.java
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.camel;
+
+import org.apache.camel.Consumer;
+import org.apache.camel.Processor;
+import org.apache.camel.Producer;
+import org.apache.camel.support.DefaultEndpoint;
+import org.apache.camel.spi.Metadata;
+import org.apache.camel.spi.UriEndpoint;
+import org.apache.camel.spi.UriParam;
+import org.apache.camel.spi.UriPath;
+
+import java.util.concurrent.ExecutorService;
+
+/**
+ * MinIO component which does bla bla.
+ *
+ * TODO: Update one line description above what the component does.
+ */
+@UriEndpoint(firstVersion = "1.0-SNAPSHOT", scheme = "MinIO", title = "MinIO", syntax="MinIO:name",

Review comment:
       Maybe `firstVersion` could be set to `3.5.0` or something like that.

##########
File path: components/camel-minio/src/main/java/org/apache/camel/MinIOEndpoint.java
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.camel;
+
+import org.apache.camel.Consumer;
+import org.apache.camel.Processor;
+import org.apache.camel.Producer;
+import org.apache.camel.support.DefaultEndpoint;
+import org.apache.camel.spi.Metadata;
+import org.apache.camel.spi.UriEndpoint;
+import org.apache.camel.spi.UriParam;
+import org.apache.camel.spi.UriPath;
+
+import java.util.concurrent.ExecutorService;
+
+/**
+ * MinIO component which does bla bla.
+ *
+ * TODO: Update one line description above what the component does.
+ */
+@UriEndpoint(firstVersion = "1.0-SNAPSHOT", scheme = "MinIO", title = "MinIO", syntax="MinIO:name",

Review comment:
       And maybe scheme could be lowercase, like "minio"




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel] DenisIstomin commented on pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

GitBox
In reply to this post by GitBox

DenisIstomin commented on pull request #3897:
URL: https://github.com/apache/camel/pull/3897#issuecomment-644160653


   Looks good @Nayananga,
   You could start to:
   1. Implement basic CRUD operations with unit\integration tests
   2. Improve docs


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel] Nayananga commented on a change in pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

GitBox
In reply to this post by GitBox

Nayananga commented on a change in pull request #3897:
URL: https://github.com/apache/camel/pull/3897#discussion_r445202915



##########
File path: components/camel-minio/src/main/java/org/apache/camel/component/minio/MinioConsumer.java
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.minio;
+
+import java.io.InputStream;
+import java.util.*;
+
+import io.minio.MinioClient;
+import io.minio.Result;
+import io.minio.errors.MinioException;
+import io.minio.messages.Item;
+import org.apache.camel.*;
+import org.apache.camel.spi.Synchronization;
+import org.apache.camel.support.ScheduledBatchPollingConsumer;
+import org.apache.camel.util.CastUtils;
+import org.apache.camel.util.IOHelper;
+import org.apache.camel.util.ObjectHelper;
+import org.apache.camel.util.URISupport;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Consumer of messages from the Minio Storage Service.
+ */
+public class MinioConsumer extends ScheduledBatchPollingConsumer {
+
+    private static final Logger LOG = LoggerFactory.getLogger(MinioConsumer.class);
+
+    private Iterator<Result<Item>> marker;
+    private transient String minioConsumerToString;
+
+    public MinioConsumer(MinioEndpoint endpoint, Processor processor) throws NoFactoryAvailableException {
+        super(endpoint, processor);
+    }
+
+    @Override
+    protected int poll() throws Exception {
+        // must reset for each poll
+        shutdownRunningTask = null;
+        pendingExchanges = 0;
+
+        MinioClient minioClient = getConfiguration().getMinioClient();
+        String objectName = getConfiguration().getObjectName();
+        String bucketName = getConfiguration().getBucketName();
+        Queue<Exchange> exchanges = null;
+
+        if (objectName != null) {
+            LOG.trace("Getting object in bucket [{}] with object name [{}]...", bucketName, objectName);
+
+            try {
+                InputStream stream = minioClient.getObject(bucketName,
+                        objectName,
+                        getConfiguration().getOffset(),
+                        getConfiguration().getLength(),
+                        getConfiguration().getServerSideEncryption());
+
+                exchanges = createExchanges(stream, objectName);
+            } catch (Exception e) {
+                LOG.trace("Failed to get object in bucket [{}] with object name [{}], Error message [{}]", bucketName, objectName, e);
+            }
+
+        } else {

Review comment:
       @oscerd @DenisIstomin What kind of approach suitable here? AWS2 S3 component have use something like this
   ```
   ListObjectsRequest.Builder listObjectsRequest = ListObjectsRequest.builder();
               listObjectsRequest.bucket(bucketName);
               listObjectsRequest.prefix(getConfiguration().getPrefix());
               listObjectsRequest.delimiter(getConfiguration().getDelimiter());
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel] oscerd commented on a change in pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

GitBox
In reply to this post by GitBox

oscerd commented on a change in pull request #3897:
URL: https://github.com/apache/camel/pull/3897#discussion_r445302954



##########
File path: components/camel-minio/src/main/java/org/apache/camel/component/minio/MinioConsumer.java
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.minio;
+
+import java.io.InputStream;
+import java.util.*;
+
+import io.minio.MinioClient;
+import io.minio.Result;
+import io.minio.errors.MinioException;
+import io.minio.messages.Item;
+import org.apache.camel.*;
+import org.apache.camel.spi.Synchronization;
+import org.apache.camel.support.ScheduledBatchPollingConsumer;
+import org.apache.camel.util.CastUtils;
+import org.apache.camel.util.IOHelper;
+import org.apache.camel.util.ObjectHelper;
+import org.apache.camel.util.URISupport;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Consumer of messages from the Minio Storage Service.
+ */
+public class MinioConsumer extends ScheduledBatchPollingConsumer {
+
+    private static final Logger LOG = LoggerFactory.getLogger(MinioConsumer.class);
+
+    private Iterator<Result<Item>> marker;
+    private transient String minioConsumerToString;
+
+    public MinioConsumer(MinioEndpoint endpoint, Processor processor) throws NoFactoryAvailableException {
+        super(endpoint, processor);
+    }
+
+    @Override
+    protected int poll() throws Exception {
+        // must reset for each poll
+        shutdownRunningTask = null;
+        pendingExchanges = 0;
+
+        MinioClient minioClient = getConfiguration().getMinioClient();
+        String objectName = getConfiguration().getObjectName();
+        String bucketName = getConfiguration().getBucketName();
+        Queue<Exchange> exchanges = null;
+
+        if (objectName != null) {
+            LOG.trace("Getting object in bucket [{}] with object name [{}]...", bucketName, objectName);
+
+            try {
+                InputStream stream = minioClient.getObject(bucketName,
+                        objectName,
+                        getConfiguration().getOffset(),
+                        getConfiguration().getLength(),
+                        getConfiguration().getServerSideEncryption());
+
+                exchanges = createExchanges(stream, objectName);
+            } catch (Exception e) {
+                LOG.trace("Failed to get object in bucket [{}] with object name [{}], Error message [{}]", bucketName, objectName, e);
+            }
+
+        } else {

Review comment:
       you could use https://docs.min.io/docs/java-client-api-reference.html#listObjects @Nayananga




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel] Nayananga commented on a change in pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

GitBox
In reply to this post by GitBox

Nayananga commented on a change in pull request #3897:
URL: https://github.com/apache/camel/pull/3897#discussion_r445307693



##########
File path: components/camel-minio/src/main/java/org/apache/camel/component/minio/MinioConsumer.java
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.minio;
+
+import java.io.InputStream;
+import java.util.*;
+
+import io.minio.MinioClient;
+import io.minio.Result;
+import io.minio.errors.MinioException;
+import io.minio.messages.Item;
+import org.apache.camel.*;
+import org.apache.camel.spi.Synchronization;
+import org.apache.camel.support.ScheduledBatchPollingConsumer;
+import org.apache.camel.util.CastUtils;
+import org.apache.camel.util.IOHelper;
+import org.apache.camel.util.ObjectHelper;
+import org.apache.camel.util.URISupport;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Consumer of messages from the Minio Storage Service.
+ */
+public class MinioConsumer extends ScheduledBatchPollingConsumer {
+
+    private static final Logger LOG = LoggerFactory.getLogger(MinioConsumer.class);
+
+    private Iterator<Result<Item>> marker;
+    private transient String minioConsumerToString;
+
+    public MinioConsumer(MinioEndpoint endpoint, Processor processor) throws NoFactoryAvailableException {
+        super(endpoint, processor);
+    }
+
+    @Override
+    protected int poll() throws Exception {
+        // must reset for each poll
+        shutdownRunningTask = null;
+        pendingExchanges = 0;
+
+        MinioClient minioClient = getConfiguration().getMinioClient();
+        String objectName = getConfiguration().getObjectName();
+        String bucketName = getConfiguration().getBucketName();
+        Queue<Exchange> exchanges = null;
+
+        if (objectName != null) {
+            LOG.trace("Getting object in bucket [{}] with object name [{}]...", bucketName, objectName);
+
+            try {
+                InputStream stream = minioClient.getObject(bucketName,
+                        objectName,
+                        getConfiguration().getOffset(),
+                        getConfiguration().getLength(),
+                        getConfiguration().getServerSideEncryption());
+
+                exchanges = createExchanges(stream, objectName);
+            } catch (Exception e) {
+                LOG.trace("Failed to get object in bucket [{}] with object name [{}], Error message [{}]", bucketName, objectName, e);
+            }
+
+        } else {
+
+            LOG.trace("Queueing objects in bucket [{}]...", bucketName);
+
+            Iterable<Result<Item>> results = minioClient.listObjects(bucketName,

Review comment:
       Hi @oscerd, here if users haven't provided any objectName then my intention is to pull all the objects in the bucket using Iterator named marker, first I would like to know is that ok to do?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel] oscerd commented on a change in pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

GitBox
In reply to this post by GitBox

oscerd commented on a change in pull request #3897:
URL: https://github.com/apache/camel/pull/3897#discussion_r445308634



##########
File path: components/camel-minio/src/main/java/org/apache/camel/component/minio/MinioConsumer.java
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.minio;
+
+import java.io.InputStream;
+import java.util.*;
+
+import io.minio.MinioClient;
+import io.minio.Result;
+import io.minio.errors.MinioException;
+import io.minio.messages.Item;
+import org.apache.camel.*;
+import org.apache.camel.spi.Synchronization;
+import org.apache.camel.support.ScheduledBatchPollingConsumer;
+import org.apache.camel.util.CastUtils;
+import org.apache.camel.util.IOHelper;
+import org.apache.camel.util.ObjectHelper;
+import org.apache.camel.util.URISupport;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Consumer of messages from the Minio Storage Service.
+ */
+public class MinioConsumer extends ScheduledBatchPollingConsumer {
+
+    private static final Logger LOG = LoggerFactory.getLogger(MinioConsumer.class);
+
+    private Iterator<Result<Item>> marker;
+    private transient String minioConsumerToString;
+
+    public MinioConsumer(MinioEndpoint endpoint, Processor processor) throws NoFactoryAvailableException {
+        super(endpoint, processor);
+    }
+
+    @Override
+    protected int poll() throws Exception {
+        // must reset for each poll
+        shutdownRunningTask = null;
+        pendingExchanges = 0;
+
+        MinioClient minioClient = getConfiguration().getMinioClient();
+        String objectName = getConfiguration().getObjectName();
+        String bucketName = getConfiguration().getBucketName();
+        Queue<Exchange> exchanges = null;
+
+        if (objectName != null) {
+            LOG.trace("Getting object in bucket [{}] with object name [{}]...", bucketName, objectName);
+
+            try {
+                InputStream stream = minioClient.getObject(bucketName,
+                        objectName,
+                        getConfiguration().getOffset(),
+                        getConfiguration().getLength(),
+                        getConfiguration().getServerSideEncryption());
+
+                exchanges = createExchanges(stream, objectName);
+            } catch (Exception e) {
+                LOG.trace("Failed to get object in bucket [{}] with object name [{}], Error message [{}]", bucketName, objectName, e);
+            }
+
+        } else {
+
+            LOG.trace("Queueing objects in bucket [{}]...", bucketName);
+
+            Iterable<Result<Item>> results = minioClient.listObjects(bucketName,

Review comment:
       Hello @Nayananga , yes it is ok. You can do that and then process the result as a batch, like the S3 AWS2 component does.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel] Nayananga commented on a change in pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

GitBox
In reply to this post by GitBox

Nayananga commented on a change in pull request #3897:
URL: https://github.com/apache/camel/pull/3897#discussion_r445310644



##########
File path: components/camel-minio/src/main/java/org/apache/camel/component/minio/MinioConsumer.java
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.minio;
+
+import java.io.InputStream;
+import java.util.*;
+
+import io.minio.MinioClient;
+import io.minio.Result;
+import io.minio.errors.MinioException;
+import io.minio.messages.Item;
+import org.apache.camel.*;
+import org.apache.camel.spi.Synchronization;
+import org.apache.camel.support.ScheduledBatchPollingConsumer;
+import org.apache.camel.util.CastUtils;
+import org.apache.camel.util.IOHelper;
+import org.apache.camel.util.ObjectHelper;
+import org.apache.camel.util.URISupport;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Consumer of messages from the Minio Storage Service.
+ */
+public class MinioConsumer extends ScheduledBatchPollingConsumer {
+
+    private static final Logger LOG = LoggerFactory.getLogger(MinioConsumer.class);
+
+    private Iterator<Result<Item>> marker;
+    private transient String minioConsumerToString;
+
+    public MinioConsumer(MinioEndpoint endpoint, Processor processor) throws NoFactoryAvailableException {
+        super(endpoint, processor);
+    }
+
+    @Override
+    protected int poll() throws Exception {
+        // must reset for each poll
+        shutdownRunningTask = null;
+        pendingExchanges = 0;
+
+        MinioClient minioClient = getConfiguration().getMinioClient();
+        String objectName = getConfiguration().getObjectName();
+        String bucketName = getConfiguration().getBucketName();
+        Queue<Exchange> exchanges = null;
+
+        if (objectName != null) {
+            LOG.trace("Getting object in bucket [{}] with object name [{}]...", bucketName, objectName);
+
+            try {
+                InputStream stream = minioClient.getObject(bucketName,
+                        objectName,
+                        getConfiguration().getOffset(),
+                        getConfiguration().getLength(),
+                        getConfiguration().getServerSideEncryption());
+
+                exchanges = createExchanges(stream, objectName);
+            } catch (Exception e) {
+                LOG.trace("Failed to get object in bucket [{}] with object name [{}], Error message [{}]", bucketName, objectName, e);
+            }
+
+        } else {
+
+            LOG.trace("Queueing objects in bucket [{}]...", bucketName);
+
+            Iterable<Result<Item>> results = minioClient.listObjects(bucketName,

Review comment:
       Thank you, is it okay to queue all the bucket objects (may take lot memory I guess) then send to  ```processBatch(CastUtils.cast(exchanges))```? because in minio I had to queue each object in a ```Queue<Object>``` until ```iterator.hasNext()``` return false




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel] oscerd commented on a change in pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

GitBox
In reply to this post by GitBox

oscerd commented on a change in pull request #3897:
URL: https://github.com/apache/camel/pull/3897#discussion_r445311790



##########
File path: components/camel-minio/src/main/java/org/apache/camel/component/minio/MinioConsumer.java
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.minio;
+
+import java.io.InputStream;
+import java.util.*;
+
+import io.minio.MinioClient;
+import io.minio.Result;
+import io.minio.errors.MinioException;
+import io.minio.messages.Item;
+import org.apache.camel.*;
+import org.apache.camel.spi.Synchronization;
+import org.apache.camel.support.ScheduledBatchPollingConsumer;
+import org.apache.camel.util.CastUtils;
+import org.apache.camel.util.IOHelper;
+import org.apache.camel.util.ObjectHelper;
+import org.apache.camel.util.URISupport;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Consumer of messages from the Minio Storage Service.
+ */
+public class MinioConsumer extends ScheduledBatchPollingConsumer {
+
+    private static final Logger LOG = LoggerFactory.getLogger(MinioConsumer.class);
+
+    private Iterator<Result<Item>> marker;
+    private transient String minioConsumerToString;
+
+    public MinioConsumer(MinioEndpoint endpoint, Processor processor) throws NoFactoryAvailableException {
+        super(endpoint, processor);
+    }
+
+    @Override
+    protected int poll() throws Exception {
+        // must reset for each poll
+        shutdownRunningTask = null;
+        pendingExchanges = 0;
+
+        MinioClient minioClient = getConfiguration().getMinioClient();
+        String objectName = getConfiguration().getObjectName();
+        String bucketName = getConfiguration().getBucketName();
+        Queue<Exchange> exchanges = null;
+
+        if (objectName != null) {
+            LOG.trace("Getting object in bucket [{}] with object name [{}]...", bucketName, objectName);
+
+            try {
+                InputStream stream = minioClient.getObject(bucketName,
+                        objectName,
+                        getConfiguration().getOffset(),
+                        getConfiguration().getLength(),
+                        getConfiguration().getServerSideEncryption());
+
+                exchanges = createExchanges(stream, objectName);
+            } catch (Exception e) {
+                LOG.trace("Failed to get object in bucket [{}] with object name [{}], Error message [{}]", bucketName, objectName, e);
+            }
+
+        } else {
+
+            LOG.trace("Queueing objects in bucket [{}]...", bucketName);
+
+            Iterable<Result<Item>> results = minioClient.listObjects(bucketName,

Review comment:
       Not all the objects in one call, you need to define a limit for the batch, like setMaxKeys in AWS2 S3 component https://github.com/apache/camel/blob/master/components/camel-aws2-s3/src/main/java/org/apache/camel/component/aws2/s3/AWS2S3Consumer.java#L88
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel] DenisIstomin commented on a change in pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

GitBox
In reply to this post by GitBox

DenisIstomin commented on a change in pull request #3897:
URL: https://github.com/apache/camel/pull/3897#discussion_r449651579



##########
File path: components/camel-minio/src/test/java/org/apache/camel/component/minio/MinioComponentTest.java
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.minio;
+
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.mock.MockEndpoint;
+import org.apache.camel.test.junit4.CamelTestSupport;
+import org.junit.Test;
+
+public class MinioComponentTest extends CamelTestSupport {
+
+    @Test
+    public void testMinio() throws Exception {
+        MockEndpoint mock = getMockEndpoint("mock:result");
+        mock.expectedMinimumMessageCount(1);
+
+        assertMockEndpointsSatisfied();

Review comment:
       Looks like that test should fail, because there are no messages being sent.

##########
File path: components/camel-minio/src/main/java/org/apache/camel/component/minio/MinioConsumer.java
##########
@@ -0,0 +1,393 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.minio;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+import java.util.*;
+
+import io.minio.DownloadObjectArgs;
+import io.minio.MinioClient;
+import io.minio.Result;
+import io.minio.errors.InvalidBucketNameException;
+import io.minio.errors.MinioException;
+import io.minio.messages.Bucket;
+import io.minio.messages.Item;
+import org.apache.camel.AsyncCallback;
+import org.apache.camel.Exchange;
+import org.apache.camel.ExtendedExchange;
+import org.apache.camel.NoFactoryAvailableException;
+import org.apache.camel.Processor;
+import org.apache.camel.spi.Synchronization;
+import org.apache.camel.support.ScheduledBatchPollingConsumer;
+import org.apache.camel.util.CastUtils;
+import org.apache.camel.util.ObjectHelper;
+import org.apache.camel.util.URISupport;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Consumer of messages from the Minio Storage Service.
+ */
+public class MinioConsumer extends ScheduledBatchPollingConsumer {
+
+    private static final Logger LOG = LoggerFactory.getLogger(MinioConsumer.class);
+
+    private Iterator<Result<Item>> marker;
+    private transient String minioConsumerToString;
+
+    public MinioConsumer(MinioEndpoint endpoint, Processor processor) throws NoFactoryAvailableException {
+        super(endpoint, processor);
+    }
+
+    @Override
+    protected int poll() throws Exception {
+        // must reset for each poll
+        shutdownRunningTask = null;
+        pendingExchanges = 0;
+
+        assert getConfiguration().getBucketName() != null;

Review comment:
       Maybe you should avoid using `assert`. It is not enabled by default. It would be better to handle that with `if` condition.

##########
File path: components/camel-minio/src/main/java/org/apache/camel/component/minio/MinioEndpoint.java
##########
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.minio;
+
+import java.io.*;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+
+import io.minio.GetObjectTagsArgs;
+import io.minio.MakeBucketArgs;
+import io.minio.MinioClient;
+import io.minio.errors.InvalidBucketNameException;
+import io.minio.messages.Tags;
+import jdk.internal.org.jline.utils.Log;
+import org.apache.camel.Category;
+import org.apache.camel.Component;
+import org.apache.camel.Consumer;
+import org.apache.camel.Exchange;
+import org.apache.camel.ExchangePattern;
+import org.apache.camel.ExtendedExchange;
+import org.apache.camel.Message;
+import org.apache.camel.Processor;
+import org.apache.camel.Producer;
+import org.apache.camel.component.minio.client.MinioClientFactory;
+import org.apache.camel.spi.Metadata;
+import org.apache.camel.spi.UriEndpoint;
+import org.apache.camel.spi.UriParam;
+import org.apache.camel.spi.UriPath;
+import org.apache.camel.support.ScheduledPollEndpoint;
+import org.apache.camel.support.SynchronizationAdapter;
+import org.apache.camel.util.ObjectHelper;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Store and retrie objects from Minio Storage Service using Minio SDK.

Review comment:
       Typo: retrieve

##########
File path: components/camel-minio/src/main/java/org/apache/camel/component/minio/MinioConsumer.java
##########
@@ -0,0 +1,393 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.minio;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+import java.util.*;
+
+import io.minio.DownloadObjectArgs;
+import io.minio.MinioClient;
+import io.minio.Result;
+import io.minio.errors.InvalidBucketNameException;
+import io.minio.errors.MinioException;
+import io.minio.messages.Bucket;
+import io.minio.messages.Item;
+import org.apache.camel.AsyncCallback;
+import org.apache.camel.Exchange;
+import org.apache.camel.ExtendedExchange;
+import org.apache.camel.NoFactoryAvailableException;
+import org.apache.camel.Processor;
+import org.apache.camel.spi.Synchronization;
+import org.apache.camel.support.ScheduledBatchPollingConsumer;
+import org.apache.camel.util.CastUtils;
+import org.apache.camel.util.ObjectHelper;
+import org.apache.camel.util.URISupport;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Consumer of messages from the Minio Storage Service.
+ */
+public class MinioConsumer extends ScheduledBatchPollingConsumer {
+
+    private static final Logger LOG = LoggerFactory.getLogger(MinioConsumer.class);
+
+    private Iterator<Result<Item>> marker;
+    private transient String minioConsumerToString;
+
+    public MinioConsumer(MinioEndpoint endpoint, Processor processor) throws NoFactoryAvailableException {
+        super(endpoint, processor);
+    }
+
+    @Override
+    protected int poll() throws Exception {
+        // must reset for each poll
+        shutdownRunningTask = null;
+        pendingExchanges = 0;
+
+        assert getConfiguration().getBucketName() != null;
+        String bucketName = getConfiguration().getBucketName();
+        MinioClient minioClient = getMinioClient();
+        String objectName = getConfiguration().getObjectName();
+        InputStream minioObject = null;
+        Queue<Exchange> exchanges = null;
+
+        if (bucketExists(minioClient, bucketName)) {
+            LOG.trace("Bucket {} exists", bucketName);
+        } else {
+            throw new InvalidBucketNameException("Bucket {} does not exists", bucketName);
+        }
+
+        if (objectName != null) {
+            LOG.trace("Getting object in bucket {} with object name {}...", bucketName, objectName);
+
+            try {
+                minioObject = getObject(bucketName, minioClient, objectName);
+                if (minioObject != null) {
+                    exchanges = createExchanges(minioObject, objectName);
+                }
+
+            } catch (Throwable e) {
+                LOG.warn("Failed to get object in bucket {} with object name {}, Error message {}", bucketName, objectName, e.getMessage());
+                throw e;
+
+            } finally {
+                //must be closed after use to release network resources.
+                try {
+                    assert minioObject != null;
+                    minioObject.close();

Review comment:
       Maybe [try-with-resources](https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html) could be used here. It would be nice to reduce nesting.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel] Nayananga commented on a change in pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

GitBox
In reply to this post by GitBox

Nayananga commented on a change in pull request #3897:
URL: https://github.com/apache/camel/pull/3897#discussion_r449760185



##########
File path: components/camel-minio/src/test/java/org/apache/camel/component/minio/MinioComponentTest.java
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.minio;
+
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.mock.MockEndpoint;
+import org.apache.camel.test.junit4.CamelTestSupport;
+import org.junit.Test;
+
+public class MinioComponentTest extends CamelTestSupport {
+
+    @Test
+    public void testMinio() throws Exception {
+        MockEndpoint mock = getMockEndpoint("mock:result");
+        mock.expectedMinimumMessageCount(1);
+
+        assertMockEndpointsSatisfied();

Review comment:
       ack, I haven't look into tests yet, I'll work on these asap :)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel] oscerd commented on a change in pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

GitBox
In reply to this post by GitBox

oscerd commented on a change in pull request #3897:
URL: https://github.com/apache/camel/pull/3897#discussion_r453283638



##########
File path: components/camel-minio/pom.xml
##########
@@ -0,0 +1,149 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+    Licensed to the Apache Software Foundation (ASF) under one or more
+    contributor license agreements.  See the NOTICE file distributed with
+    this work for additional information regarding copyright ownership.
+    The ASF licenses this file to You under the Apache License, Version 2.0
+    (the "License"); you may not use this file except in compliance with
+    the License.  You may obtain a copy of the License at
+
+         http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+
+-->
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+
+    <parent>
+        <groupId>org.apache.camel</groupId>
+        <artifactId>components</artifactId>
+        <version>3.5.0-SNAPSHOT</version>
+    </parent>
+
+    <artifactId>camel-minio</artifactId>
+    <packaging>jar</packaging>
+
+    <name>Camel :: MinIO</name>
+    <description>A Camel MinIO cloud storage server Component</description>
+
+    <properties>
+        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
+        <camel.osgi.export.pkg>org.apache.camel.component.minio.*</camel.osgi.export.pkg>
+        <camel.osgi.export.service>
+            org.apache.camel.spi.ComponentResolver;component=minio
+        </camel.osgi.export.service>
+    </properties>
+
+    <dependencies>
+        <!-- camel -->
+        <dependency>
+            <groupId>org.apache.camel</groupId>
+            <artifactId>camel-support</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>io.minio</groupId>
+            <artifactId>minio</artifactId>
+            <version>7.1.0</version>

Review comment:
       This should go as property in parent pom

##########
File path: components/camel-minio/pom.xml
##########
@@ -0,0 +1,149 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+    Licensed to the Apache Software Foundation (ASF) under one or more
+    contributor license agreements.  See the NOTICE file distributed with
+    this work for additional information regarding copyright ownership.
+    The ASF licenses this file to You under the Apache License, Version 2.0
+    (the "License"); you may not use this file except in compliance with
+    the License.  You may obtain a copy of the License at
+
+         http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+
+-->
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+
+    <parent>
+        <groupId>org.apache.camel</groupId>
+        <artifactId>components</artifactId>
+        <version>3.5.0-SNAPSHOT</version>
+    </parent>
+
+    <artifactId>camel-minio</artifactId>
+    <packaging>jar</packaging>
+
+    <name>Camel :: MinIO</name>
+    <description>A Camel MinIO cloud storage server Component</description>
+
+    <properties>
+        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
+        <camel.osgi.export.pkg>org.apache.camel.component.minio.*</camel.osgi.export.pkg>
+        <camel.osgi.export.service>
+            org.apache.camel.spi.ComponentResolver;component=minio
+        </camel.osgi.export.service>
+    </properties>
+
+    <dependencies>
+        <!-- camel -->
+        <dependency>
+            <groupId>org.apache.camel</groupId>
+            <artifactId>camel-support</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>io.minio</groupId>
+            <artifactId>minio</artifactId>
+            <version>7.1.0</version>
+        </dependency>
+
+        <!-- logging -->
+        <dependency>
+            <groupId>org.apache.logging.log4j</groupId>
+            <artifactId>log4j-api</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.logging.log4j</groupId>
+            <artifactId>log4j-core</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.logging.log4j</groupId>
+            <artifactId>log4j-slf4j-impl</artifactId>
+            <scope>test</scope>
+        </dependency>
+
+        <!-- testing -->
+        <dependency>
+            <groupId>org.apache.camel</groupId>
+            <artifactId>camel-test</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.camel</groupId>

Review comment:
       I don't think aws2-s3 is needed




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel] Nayananga commented on a change in pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

GitBox
In reply to this post by GitBox

Nayananga commented on a change in pull request #3897:
URL: https://github.com/apache/camel/pull/3897#discussion_r456499948



##########
File path: components/camel-minio/pom.xml
##########
@@ -0,0 +1,149 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+    Licensed to the Apache Software Foundation (ASF) under one or more
+    contributor license agreements.  See the NOTICE file distributed with
+    this work for additional information regarding copyright ownership.
+    The ASF licenses this file to You under the Apache License, Version 2.0
+    (the "License"); you may not use this file except in compliance with
+    the License.  You may obtain a copy of the License at
+
+         http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+
+-->
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+
+    <parent>
+        <groupId>org.apache.camel</groupId>
+        <artifactId>components</artifactId>
+        <version>3.5.0-SNAPSHOT</version>
+    </parent>
+
+    <artifactId>camel-minio</artifactId>
+    <packaging>jar</packaging>
+
+    <name>Camel :: MinIO</name>
+    <description>A Camel MinIO cloud storage server Component</description>
+
+    <properties>
+        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
+        <camel.osgi.export.pkg>org.apache.camel.component.minio.*</camel.osgi.export.pkg>
+        <camel.osgi.export.service>
+            org.apache.camel.spi.ComponentResolver;component=minio
+        </camel.osgi.export.service>
+    </properties>
+
+    <dependencies>
+        <!-- camel -->
+        <dependency>
+            <groupId>org.apache.camel</groupId>
+            <artifactId>camel-support</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>io.minio</groupId>
+            <artifactId>minio</artifactId>
+            <version>7.1.0</version>

Review comment:
       @oscerd Thank you for the suggestion. Would you mind telling me where exactly this property should go? is it pom.xml in component root directory?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel] oscerd commented on a change in pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

GitBox
In reply to this post by GitBox

oscerd commented on a change in pull request #3897:
URL: https://github.com/apache/camel/pull/3897#discussion_r456501227



##########
File path: components/camel-minio/pom.xml
##########
@@ -0,0 +1,149 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+    Licensed to the Apache Software Foundation (ASF) under one or more
+    contributor license agreements.  See the NOTICE file distributed with
+    this work for additional information regarding copyright ownership.
+    The ASF licenses this file to You under the Apache License, Version 2.0
+    (the "License"); you may not use this file except in compliance with
+    the License.  You may obtain a copy of the License at
+
+         http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+
+-->
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+
+    <parent>
+        <groupId>org.apache.camel</groupId>
+        <artifactId>components</artifactId>
+        <version>3.5.0-SNAPSHOT</version>
+    </parent>
+
+    <artifactId>camel-minio</artifactId>
+    <packaging>jar</packaging>
+
+    <name>Camel :: MinIO</name>
+    <description>A Camel MinIO cloud storage server Component</description>
+
+    <properties>
+        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
+        <camel.osgi.export.pkg>org.apache.camel.component.minio.*</camel.osgi.export.pkg>
+        <camel.osgi.export.service>
+            org.apache.camel.spi.ComponentResolver;component=minio
+        </camel.osgi.export.service>
+    </properties>
+
+    <dependencies>
+        <!-- camel -->
+        <dependency>
+            <groupId>org.apache.camel</groupId>
+            <artifactId>camel-support</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>io.minio</groupId>
+            <artifactId>minio</artifactId>
+            <version>7.1.0</version>

Review comment:
       It should go in parent/pom.xml




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel] omarsmak commented on pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

GitBox
In reply to this post by GitBox

omarsmak commented on pull request #3897:
URL: https://github.com/apache/camel/pull/3897#issuecomment-662320735


   Hey @Nayananga I assume this PR is still WIP?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel] Nayananga commented on pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

GitBox
In reply to this post by GitBox

Nayananga commented on pull request #3897:
URL: https://github.com/apache/camel/pull/3897#issuecomment-663611575


   @zregvart @oscerd @DenisIstomin I would like to request to merge this PR as the first version of the component. Please let me know if there are necessary changes to do before that.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel] Nayananga commented on pull request #3897: CAMEL-13934 camel-minio - Component to store/load files from blob store

GitBox
In reply to this post by GitBox

Nayananga commented on pull request #3897:
URL: https://github.com/apache/camel/pull/3897#issuecomment-663884701


   > Hey @Nayananga I assume this PR is still WIP?
   
   Hi @omarsmak, sorry I just saw this, I guess this is ready to review


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


123