[GitHub] [camel-k] kameshsampath opened a new issue #1172: Knative integration takes too long to make the service ready

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-k] kameshsampath opened a new issue #1172: Knative integration takes too long to make the service ready

GitBox
kameshsampath opened a new issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172
 
 
   When trying to create a Knative integration I observe that the Knative service is taking too long to get to ready state and sometimes its struck in `RevisionMissing` state.  This happens in both `--dev` mode as well as in `production` mode. This is highly inconsistent as sometimes it refreshes the state of the service soon but after few runs its starts to get hung .
   
   It is also noted it takes very long time to trigger the service deployment after we call `kamel run`, wondering something to do with Knative Serving reconcilation ? There should be a way to force the knative recocilation, so that new reivision is updated with Ready state as soon as possible.
   
   Example integation:
   
   ```yaml
   - from:
       uri: "knative:endpoint/echoer"
       steps:
         - log:
             message: "Got Message: ${body}"
         - convert-body: "java.lang.String"
         - choice:
             when:
               - simple: "${body} != null && ${body.length} > 0"
                 steps:
                   - set-body:
                       simple: "${body.toUpperCase()}"
                   - set-header:
                       name: ContentType
                       simple: text/plain
                   - log:
                       message: "${body}"
             otherwise:
               steps:
                 - set-body:
                     constant: "no body received"
                 - set-header:
                     name: ContentType
                     simple: text/plain
                 - log:
                     message: "Otherwise::${body}"
   
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-k] kameshsampath commented on issue #1172: Knative integration takes too long to make the service ready

GitBox
kameshsampath commented on issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172#issuecomment-568327565
 
 
   Sample output of `ksvc`
   
   ```
   NAME     URL                                         LATESTCREATED   LATESTREADY   READY     REASON
   echoer   http://echoer.knativetutorial.example.com   echoer-kpcmz                  Unknown   RevisionMissing
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-k] kameshsampath edited a comment on issue #1172: Knative integration takes too long to make the service ready

GitBox
In reply to this post by GitBox
kameshsampath edited a comment on issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172#issuecomment-568327565
 
 
   Sample output of `ksvc`
   
   ```
   NAME     URL                                         LATESTCREATED   LATESTREADY   READY     REASON
   echoer   http://echoer.knativetutorial.example.com   echoer-kpcmz                  Unknown   RevisionMissing
   ```
   
   But when you see the pods of the namespace you will notice the Knative service pod is up
   
   ```▶ kgpo
   NAME                                       READY   STATUS      RESTARTS   AGE
   camel-k-cache                              0/1     Completed   0          10h
   camel-k-kit-bnvnkhccmvshuqdcf8ng           0/1     Completed   0          11h
   camel-k-kit-bnvnqlccmvshuqdcf8o0           0/1     Completed   0          11h
   camel-k-kit-bnvodbccmvshuqdcf8og           0/1     Completed   0          10h
   camel-k-kit-bnvodbccmvshuqdcf8og-builder   0/1     Completed   0          10h
   camel-k-kit-bo0102kcmvshuqdcf8p0           0/1     Completed   0          62m
   camel-k-kit-bo0102kcmvshuqdcf8p0-builder   0/1     Completed   0          62m
   camel-k-operator-84d7896b68-6gv52          1/1     Running     0          11h
   echoer-kpcmz-deployment-b9f85668c-ddzs4    2/2     Running     0          63s```
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-k] kameshsampath edited a comment on issue #1172: Knative integration takes too long to make the service ready

GitBox
In reply to this post by GitBox
kameshsampath edited a comment on issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172#issuecomment-568327565
 
 
   Sample output of `ksvc`
   
   ```
   NAME     URL                                         LATESTCREATED   LATESTREADY   READY     REASON
   echoer   http://echoer.knativetutorial.example.com   echoer-kpcmz                  Unknown   RevisionMissing
   ```
   
   But when you see the pods of the namespace you will notice the Knative service pod is up
   
   ```
   ▶ kgpo
   NAME                                       READY   STATUS      RESTARTS   AGE
   camel-k-cache                              0/1     Completed   0          10h
   camel-k-kit-bnvnkhccmvshuqdcf8ng           0/1     Completed   0          11h
   camel-k-kit-bnvnqlccmvshuqdcf8o0           0/1     Completed   0          11h
   camel-k-kit-bnvodbccmvshuqdcf8og           0/1     Completed   0          10h
   camel-k-kit-bnvodbccmvshuqdcf8og-builder   0/1     Completed   0          10h
   camel-k-kit-bo0102kcmvshuqdcf8p0           0/1     Completed   0          62m
   camel-k-kit-bo0102kcmvshuqdcf8p0-builder   0/1     Completed   0          62m
   camel-k-operator-84d7896b68-6gv52          1/1     Running     0          11h
   echoer-kpcmz-deployment-b9f85668c-ddzs4    2/2     Running     0          63s
   ```
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-k] kameshsampath commented on issue #1172: Knative integration takes too long to make the service ready

GitBox
In reply to this post by GitBox
kameshsampath commented on issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172#issuecomment-568837966
 
 
   I also observe that in these cases the `containerConcurrency` of the ksvc and its related configuration is set to `0`.  Wondering this is causing the revision not to become ready immediately ??
   
   CC: @nicolaferraro  

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-k] nicolaferraro commented on issue #1172: Knative integration takes too long to make the service ready

GitBox
In reply to this post by GitBox
nicolaferraro commented on issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172#issuecomment-569226494
 
 
   I've hit the issue of services taking a long time to deploy on minikube, I've opened this: https://github.com/knative/serving/issues/5802, but no luck.
   I'm not experiencing the issue anymore after I started to follow the exact procedure when installing Knative, like:
   - using the suggested minikube/kube version
   - using the exact istio version, with no injection
   - installing the full monitoring stack
   
   I've also noticed that sometimes the service state is not aligned with the pod state but didn't investigate becaus the service was running in my case. Will try to figure out.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-k] kameshsampath commented on issue #1172: Knative integration takes too long to make the service ready

GitBox
In reply to this post by GitBox
kameshsampath commented on issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172#issuecomment-569380759
 
 
   > using the suggested minikube/kube version
   
   you mean as per Knative upstream docs ? I am doing exactly the same my start script for minikube is
   
   ```shell
   #!/bin/bash
   
   set -eu
   
   PROFILE_NAME=${PROFILE_NAME:-knativetutorial}
   VM_DRIVER=${VM_DRIVER:-virtualbox}
   MEMORY=${MEMORY:-8192}
   CPUS=${CPUS:-6}
   
   EXTRA_CONFIG="apiserver.enable-admission-plugins=\
   LimitRanger,\
   NamespaceExists,\
   NamespaceLifecycle,\
   ResourceQuota,\
   ServiceAccount,\
   DefaultStorageClass,\
   MutatingAdmissionWebhook"
   
   minikube start --memory=$MEMORY --cpus=$CPUS \
     --kubernetes-version=v1.14.0 \
     --vm-driver="$VM_DRIVER" \
     --disk-size=50g \
     --extra-config="$EXTRA_CONFIG" \
     --insecure-registry='10.0.0.0/24'
   ```
   
   I usually the problem more often when you try updating the service with updates either via redeploy or  in `--dev` mode
   
   > using the exact istio version, with no injection
   
   yes injection is diabled and I use the `istio-lean` install
   
   > installing the full monitoring stack
   
   Dont know why this is needed ? With 8GB RAM this will not make us to run any other applications :(
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-k] nak3 commented on issue #1172: Knative integration takes too long to make the service ready

GitBox
In reply to this post by GitBox
nak3 commented on issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172#issuecomment-583941124
 
 
   Can we more narrow down what component/event takes long time? For example:
   * Examine event logs during the deploy - `kubect get event -w`.
   * Investigate controller logs - `kubectl -n knative-serving logs controller-xxx -c controller`. (like https://github.com/knative/serving/issues/5802 did.)
   
   Also, it would be worth to look into some suspicious stuff like:
   
   * CPU/memory usage on the host to verify no problem - `$ top`.
   * The status of sub resource in the cluster. - eg. the log says `RevisionMissing`, but `kubect get rev -o yaml` shows anything?)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-k] lburgazzoli commented on issue #1172: Knative integration takes too long to make the service ready

GitBox
In reply to this post by GitBox
lburgazzoli commented on issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172#issuecomment-584521454
 
 
   releated to #1172

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-k] lburgazzoli commented on issue #1172: Knative integration takes too long to make the service ready

GitBox
In reply to this post by GitBox
lburgazzoli commented on issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172#issuecomment-584521888
 
 
   @kameshsampath @nak3 I remember I had some issues in the past similar to this one and were related to the default requests that knative sets-up. Lowering them did fix the issue.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-k] nicolaferraro commented on issue #1172: Knative integration takes too long to make the service ready

GitBox
In reply to this post by GitBox
nicolaferraro commented on issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172#issuecomment-586296568
 
 
   Fixed by #1259 (which reference images directly via digest, so that Knative does not try to contact the registry to find it)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [camel-k] nicolaferraro closed issue #1172: Knative integration takes too long to make the service ready

GitBox
In reply to this post by GitBox
nicolaferraro closed issue #1172: Knative integration takes too long to make the service ready
URL: https://github.com/apache/camel-k/issues/1172
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services