Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG OpenCTI Connector is not working #2122

Open
CyberAbwehr opened this issue Feb 9, 2024 · 12 comments
Open

BUG OpenCTI Connector is not working #2122

CyberAbwehr opened this issue Feb 9, 2024 · 12 comments
Labels
bug Something isn't working

Comments

@CyberAbwehr
Copy link

CyberAbwehr commented Feb 9, 2024

IntelOWL: v5.2.3
OPENCTI: 5.12.14

Cant send observable to OPENCTI
"OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration..."

OPENCTI Secrets are set:
api_key_name
url_key_name

Plugins -> Connectors -> OpenCTI -> perform health check
it is up and running

docker logs intelowl_celery_worker_default

INFO:intel_owl.tasks:Configuring plugin OpenCTI for job 8 with task 3093249f-f5b9-4e2c-8501-9b41924cd6db
INFO:intel_owl.tasks:Starting plugin OpenCTI for job 8 with task 3093249f-f5b9-4e2c-8501-9b41924cd6db
INFO:api_app.connectors_manager.classes:STARTED connector: (OpenCTI, job: #8)
INFO:api_app.connectors_manager.classes:Running connector OpenCTI even if job status is reported_with_fails becauserun on failure is set
INFO:pycti.entities:Listing Threat-Actors with filters null.
ERROR:pycti.api:Unknown type "ThreatActorsFiltering". Did you mean "ThreatActorsOrdering", "ThreatActorConnection", "ThreatActorEdge", "ThreatActorGroup", or "ThreatActorGroupEdge"?
ERROR:pycti.api:{'name': 'Unknown type "ThreatActorsFiltering". Did you mean "ThreatActorsOrdering", "ThreatActorConnection", "ThreatActorEdge", "ThreatActorGroup", or "ThreatActorGroupEdge"?', 'message': 'Unknown type "ThreatActorsFiltering". Did you mean "ThreatActorsOrdering", "ThreatActorConnection", "ThreatActorEdge", "ThreatActorGroup", or "ThreatActorGroupEdge"?'}
Traceback (most recent call last):
  File "/opt/deploy/intel_owl/api_app/classes.py", line 190, in start
    _result = self.run()
  File "/opt/deploy/intel_owl/api_app/connectors_manager/connectors/opencti.py", line 110, in run
    self.opencti_instance = pycti.OpenCTIApiClient(
  File "/usr/local/lib/python3.9/site-packages/pycti/api/opencti_api_client.py", line 217, in __init__
    raise ValueError(
ValueError: OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration...
ERROR:api_app.classes:(OpenCTI, job: #8). Unexpected error: 'OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration...'
Traceback (most recent call last):
  File "/opt/deploy/intel_owl/api_app/classes.py", line 190, in start
    _result = self.run()
  File "/opt/deploy/intel_owl/api_app/connectors_manager/connectors/opencti.py", line 110, in run
    self.opencti_instance = pycti.OpenCTIApiClient(
  File "/usr/local/lib/python3.9/site-packages/pycti/api/opencti_api_client.py", line 217, in __init__
    raise ValueError(
ValueError: OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration...
INFO:api_app.connectors_manager.classes:FINISHED connector: (OpenCTI, job: #8)
INFO:celery.app.trace:Task run_plugin[3093249f-f5b9-4e2c-8501-9b41924cd6db] succeeded in 0.09238294186070561s: None

Testing on the same system with pycti:

root@intelowl:/opt/pycti# python3 test.py
INFO:api:Health check (me)...
INFO:api:Creating Identity
INFO:api:Creating Identity
INFO:api:Creating stix_core_relationship
{'id': 'dcb77b4f-5ddc-45d6-8fd8-9d139bedc817', 'standard_id': 'relationship--65098457-d570-4bdf-9cb3-831e17ef125f', 'entity_type': 'part-of', 'parent_types': ['basic-relationship', 'stix-relationship', 'stix-core-relationship'], 'createdById': None}

pycti-version 5.12.14

Environment

OS: Ubuntu 22.04 LTS
IntelOwl version: v5.2.3
YETI: 2.1.3

What did you expect to happen

Write IOCs to OPENCTI
How to reproduce your issue
Error messages and logs
"OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration..."

@CyberAbwehr CyberAbwehr added the bug Something isn't working label Feb 9, 2024
@mlodic
Copy link
Member

mlodic commented Feb 9, 2024

hey!

some things:

  • the health check does verify only that the service is UP via a HEAD request, it serves as a way to tell you that it can connect to it but if the integration is broken, it still is.
  • we have received several other reports of problems with OpenCTI integration. Because of that we have a specific section of the documentation which explains the problem and how to solve it: https://intelowl.readthedocs.io/en/latest/Advanced-Configuration.html#opencti

@CyberAbwehr
Copy link
Author

Still the same issue. ;-)

root@intelowl:/opt/IntelOwl# ./start test build --pycti-version 5.12.14
grep: docker/.env.start.test: No such file or directory
./start: line 92: export: `': not a valid identifier
WARN[0000] The "REPO_DOWNLOADER_ENABLED" variable is not set. Defaulting to a blank string.
[+] Building 14.0s (27/27) FINISHED docker:default
=> [uwsgi internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 3.31kB 0.0s
=> [uwsgi internal] load .dockerignore 0.0s
=> => transferring context: 306B 0.0s
=> [uwsgi internal] load metadata for docker.io/library/python:3.9.16 1.1s
=> [uwsgi internal] load metadata for docker.io/library/node:lts-alpine3.18 1.1s
=> [uwsgi backend-build 1/10] FROM docker.io/library/python:3.9.16@sha256:603ac689b89c2a59791a4e7cd3d727f2a673ac3df02dabbd97b0d85bb1eca4e7 0.0s
=> [uwsgi internal] load build context 0.1s
=> => transferring context: 861.58kB 0.1s
=> [uwsgi frontend-build 1/5] FROM docker.io/library/node:lts-alpine3.18@sha256:3aae0ea51b2952660b4b65988963b78b269cf84cc7f36f208462601a12e1531a 0.0s
=> CACHED [uwsgi frontend-build 2/5] COPY frontend/ . 0.0s
=> CACHED [uwsgi frontend-build 3/5] COPY docker/.env .env.local 0.0s
=> CACHED [uwsgi frontend-build 4/5] RUN npm install npm@latest --location=global && npm install && PUBLIC_URL=/static/reactapp/ npm run build 0.0s
=> CACHED [uwsgi backend-build 2/10] RUN mkdir -p /var/log/intel_owl /var/log/intel_owl/django /var/log/intel_owl/uwsgi /opt/deploy/files 0.0s
=> CACHED [uwsgi backend-build 3/10] RUN apt-get update && apt-get install -y --no-install-recommends apt-utils libsasl2-dev libssl-dev netcat 0.0s
=> CACHED [uwsgi backend-build 4/10] RUN cpan -T Email::Outlook::Message 0.0s
=> CACHED [uwsgi backend-build 5/10] COPY requirements/project-requirements.txt /opt/deploy/intel_owl/project-requirements.txt 0.0s
=> CACHED [uwsgi backend-build 6/10] COPY requirements/certego-requirements.txt /opt/deploy/intel_owl/certego-requirements.txt 0.0s
=> CACHED [uwsgi backend-build 7/10] WORKDIR /opt/deploy/intel_owl 0.0s
=> CACHED [uwsgi backend-build 8/10] RUN pip3 install --no-cache-dir --compile -r project-requirements.txt && pip3 install --no-cache-dir elastic 0.0s
=> [uwsgi backend-build 9/10] COPY . /opt/deploy/intel_owl 0.2s
=> [uwsgi backend-build 10/10] RUN touch /var/log/intel_owl/django/api_app.log /var/log/intel_owl/django/api_app_errors.log && touch /var/log/int 11.1s
=> [uwsgi stage-2 1/1] COPY --from=frontend-build /build /var/www/reactapp 0.2s
=> [uwsgi] exporting to image 0.3s
=> => exporting layers 0.3s
=> => writing image sha256:83611a2a4ef85aece736e1773406c0f1df54c68831fc35702f1edbd64370fe59 0.0s
=> => naming to docker.io/intelowlproject/intelowl:test 0.0s
=> [nginx internal] load .dockerignore 0.0s
=> => transferring context: 306B 0.0s
=> [nginx internal] load build definition from Dockerfile_nginx 0.0s
=> => transferring dockerfile: 430B 0.0s
=> [nginx internal] load metadata for docker.io/library/nginx:1.25-alpine 1.1s
=> [nginx 1/2] FROM docker.io/library/nginx:1.25-alpine@sha256:f2802c2a9d09c7aa3ace27445dfc5656ff24355da28e7b958074a0111e3fc076 0.0s
=> CACHED [nginx 2/2] RUN rm /var/log/nginx/access.log /var/log/nginx/error.log && touch /var/log/nginx/access.log /var/log/nginx/error.log 0.0s
=> [nginx] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:7bf491e8ac1fccd5216fde650de1fb53e97baa10e68b890c2c9c77bf6ae97e4f 0.0s
=> => naming to docker.io/intelowlproject/intelowl_nginx:test 0.0s

intelowl_celery_worker_default | INFO:pycti.entities:Listing Threat-Actors with filters null.
intelowl_celery_worker_default | ERROR:pycti.api:Unknown type "ThreatActorsFiltering". Did you mean "ThreatActorsOrdering", "ThreatActorConnection", "ThreatActorEdge", "ThreatActorGroup", or "ThreatActorGroupEdge"?
intelowl_celery_worker_default | ERROR:pycti.api:{'name': 'Unknown type "ThreatActorsFiltering". Did you mean "ThreatActorsOrdering", "ThreatActorConnection", "ThreatActorEdge", "ThreatActorGroup", or "ThreatActorGroupEdge"?', 'message': 'Unknown type "ThreatActorsFiltering". Did you mean "ThreatActorsOrdering", "ThreatActorConnection", "ThreatActorEdge", "ThreatActorGroup", or "ThreatActorGroupEdge"?'}
intelowl_celery_worker_default | Traceback (most recent call last):
intelowl_celery_worker_default | File "/opt/deploy/intel_owl/api_app/classes.py", line 190, in start
intelowl_celery_worker_default | _result = self.run()
intelowl_celery_worker_default | File "/opt/deploy/intel_owl/api_app/connectors_manager/connectors/opencti.py", line 110, in run
intelowl_celery_worker_default | self.opencti_instance = pycti.OpenCTIApiClient(
intelowl_celery_worker_default | File "/usr/local/lib/python3.9/site-packages/pycti/api/opencti_api_client.py", line 217, in init
intelowl_celery_worker_default | raise ValueError(
intelowl_celery_worker_default | ValueError: OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration...
intelowl_celery_worker_default | ERROR:api_app.classes:(OpenCTI, job: #10). Unexpected error: 'OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration...'
intelowl_celery_worker_default | Traceback (most recent call last):
intelowl_celery_worker_default | File "/opt/deploy/intel_owl/api_app/classes.py", line 190, in start
intelowl_celery_worker_default | _result = self.run()
intelowl_celery_worker_default | File "/opt/deploy/intel_owl/api_app/connectors_manager/connectors/opencti.py", line 110, in run
intelowl_celery_worker_default | self.opencti_instance = pycti.OpenCTIApiClient(
intelowl_celery_worker_default | File "/usr/local/lib/python3.9/site-packages/pycti/api/opencti_api_client.py", line 217, in init
intelowl_celery_worker_default | raise ValueError(
intelowl_celery_worker_default | ValueError: OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration...
intelowl_celery_worker_default | INFO:api_app.connectors_manager.classes:FINISHED connector: (OpenCTI, job: #10)
intelowl_celery_worker_default | INFO:celery.app.trace:Task run_plugin[675c63cd-64c8-4ba9-964a-df80f0a39385] succeeded in 0.22445168299600482s: None
intelowl_celery_worker_default | INFO:api_app.models:[STARTING] set_final_status for <-- Job(#10, "145.239.253.78").
intelowl_celery_worker_default | INFO:api_app.models:[REPORT] Job(#10, "145.239.253.78"), status:connectors_completed, reports:{'success': 1, 'failed': 1, 'all': 2, 'pending': 0, 'killed': 0, 'running': 0}
intelowl_celery_worker_default | INFO:api_app.models:<Job: Job(#10, "145.239.253.78")> setting status to reported_with_fails
intelowl_celery_worker_default | INFO:celery.app.trace:Task job_set_final_status[87f22d00-6ec2-4d59-af27-b8fb9d807839] succeeded in 0.008970469236373901s: None
intelowl_celery_worker_default | INFO:celery.app.trace:Task intel_owl.tasks.update[48c6357c-8dc0-421c-95de-df81b0ab0bf8] succeeded in 0.00442303204908967s: None

@mlodic
Copy link
Member

mlodic commented Feb 9, 2024

ouch. I don't know so, it seems this connector must require additional testing....similar problem was arise here: #1730....

Are you sure to have properly set your SSL certificate, you URL and your token?

@CyberAbwehr
Copy link
Author

Secrets
Connector - OpenCTI - url_key_name - http://192.168.1.43:8080
Connector - OpenCTI - api_key_name - ************************

Parameters
Connector - OpenCTI - ssl_verify - false

@CyberAbwehr
Copy link
Author

Cant find the value "ThreatActorsFiltering"

OpenCTI API graphql OpenCTI API graphql 2

@mlodic
Copy link
Member

mlodic commented Feb 9, 2024

I don't have an OpenCTI instance available to do some tests, if you could help me to find which is the problem...

Then, this is something not referred by our connector so it is likely a bug of the pycti library itself instead of a bug of the IntelOwl's connector

@CyberAbwehr
Copy link
Author

CyberAbwehr commented Feb 9, 2024 via email

@mlodic
Copy link
Member

mlodic commented May 14, 2024

do you still have problems with opencti?

@CyberAbwehr
Copy link
Author

Need to update my systems, after that i will check.

@farwarx

This comment was marked as spam.

@farwarx

This comment was marked as spam.

@mlodic
Copy link
Member

mlodic commented Jun 2, 2024

you are talking about another project...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants