How can I get input from command python3 mycode.py <PATH>?
I google it, but all of the results are using input() or raw_input() from command after running python3 mycode.py.

How can I get input when I run python3 mycode.py <PATH> at same time without typing input on command again?

Thank,you!

Hello TAs,
I managed to change fields.hostname after reading #16

When I restart services of winlogbeat and packetbeat,
Kibana create a new index pattern, yet the number of the fields seems different.

My winlogbeat.yml

#---------fields---------
fields:
hostname: _309551108
#----------kibana---------
setup.kibana:
host: "192.168.66.1:5601"
username: "admin"
username: "admin"
#------Logstash Output ----------
output.logstash:
hosts: ["192.168.66.1:5044"]
username: "admin"
password: "admin"

my first winlogbeat patterns: 679 fields

my first packetbeat patterns: 93 fields

After restarting service , new index patterns: 340 fields

(changing hostname of both winlogbeat and packet)

The Problems

1. the old winlogbeat pattern still receive some winlogbeat data(the hostname is still unknown)
   

2. new index patterns seems to only accept packetbeat's data(the hostname shows correctly)
   

My questions

My question is as follows:

1. Do winlogbeat & packetbeat's data can send to the same index pattern? If so, how to distinguish them?
2. Why the hostname is still unknown after changing fields in winlogbeat.yml?
3. the cmd ./winlogbeat.exe setup -e will prompt out error(like the pictures below), but cmd ./winlogbeat.exe -e or "Start-Service winlogbeat" works properly, did the system reject winlogbeat.yml after changing the hostname or did anything I miss?
   

Reply or discussion is appreciated, thanks!

As title,
I tyied to use pd.read_json(json_file,lines=True), but it raise Memory error
because I need to extract key,value from the json_file
so, I used json.loads(line) (line is one line in json_file) to read one by one, and extract the key/value I need.
it work, but I found it too slow to read(40k lines/5 mins).

Is there any better way to read?
thanks!

Dear TAs,
I just finish my setting following the instruction you gave in previous issue question https://www.elastic.co/guide/en/beats/winlogbeat/7.11/winlogbeat-installation-configuration.html
Unfortunately, although I can successfully log into Kibana but there is no data related to Winlogbeat

my Kibana now looks like this

I tried to set index pattern winlogbeat-* as the instruction but there's no such data

My winlogbeat.yml

# =================================== Kibana ===================================
setup.kibana:
  # Kibana Host
  host: "192.168.66.1:5601"
  username: "admin"
  password: "admin"

and I commented the Elasticsearch Output

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["192.168.66.1:5044"]

My question is

1. Do I need to do any command on the ubuntu side or I just need to open and sign in?
2. Do I need to change any config and yml file in ubuntu side?
3. Since I also set the logging.files in winlogbeat.yml , I got the log and the error message is like this

I cannot connent to Logstash but I don't know why

I think I must missed something, sorry for bothering you🥺 🥺

As title,

and will each attack appear exactly once?

Thanks!

請問要如何正確設定這次作業的group policy editor?
我感覺是在user configuration->windows settings->scripts(logon/logoff)裡輸入想要觀察的指令(新增刪除檔案 開啟關閉小算盤)
但是我仍在在event viewer與kibana上沒看到相關log

助教您好，
我看了前面group policy setting的問題後有照著助教給的方向改動Audit Policy, 如此圖上方已經換成success,Failure
但是我還是收不到相應的log資料 ex.logon success(4624)

上網查了資料不確定是否需要用到Group Policy Management來額外做設定，但是因為電腦是workgroup所以沒辦法使用，
想請問我現在的方向是對的嗎？

Can classmates or TAs give me some suggestions or tips for project2?
I think I am in trouble T^T

Currently I have tried to import log into elasticsearch by using elasticdump
But I have no idea how to visualize and analyze in KIBANA

And I didn’t have any machine learning experience before
A little difficult for me
There are too many packages, but I don’t know which one is suitable for this job
(multi-class classification?)

So I came up to ask for help, hoping someone could give me directions to complete the assignment QQ

Thanks

Hi TA, I would like to ask the Question of environment:
I used the VM released by TA, but it turned out that whenever I started up the Windows VM, the machine just kept lagging and restarting itself again and again. So I'm wondering if it's OK for me to install a Windows 10 VM by myself? Or there is some configuration already setted up in the one released by TA., thank you.

Dear TAs:

What I have done

• connect to the kibana on my virtual machine
  

• According to the issues, I did port forwarding, but I'm not sure it is correct or not.

Problems

As the title

I'm wondering that I didn't correct winlogbeat.yml as the right way, or my window just can't connect to my virtual box.
Thanks in advance. 😭

Question about Winlogbeat & Packetbeat

• When I tried to start .\winlogbeat.exe setup -e & .\packetbeat.exe setup -e in Powershell (select run as Administrator), I would get an error message. ("error connecting to Elasticsearch at http://ubuntu_id:9200") I can't find new log in Kibana so I guess the reason behind is this issue. Does any suggestion for this issue??

• I tried to use ping (windows->linux & linux->windows) to check and I found only linux can ping to windows. I got a research about this issue and they said Network setup of NAT that they can't ping to each other. But using the Bridge, they can ping to each other.

I can't import json file to ES using elasticdump, it keeps showing socket hang up error.
What did I do wrong?
(I'm using the ES given by TA in HW01)

As title,

Also, must we pass <FILE_PATH> via command line argument?

Thanks!!

$ sudo sysctl -w vm.max_map_count=262144

I saw this command on the slide, why is this command needed, and why do we need to execute it every time after rebooting?

Hi TA,
I have two questions to ask:
1. I want to set up the configuration of the winlogbeat and I am on step 3, but it says "more than one configured accessing
output, what is the problem I might have met?
2. Is the elastic.zip file need to be launched on the Ubuntu VM? If so, I am not sure how to install the docker first before
docker-compose up

請問助教，
1.我發生了圖片右下角的error，請問要怎麼解決呢

2.請問第a和b小題是要從哪裡登入登出呢?

I have a problem about different format of event action!
First, it appear like this

Then, after I reboot both Windows and Ubuntu, it somehow change to this

However, there's no different between the code and setting.
Furthermore, some fields are't available now! e.g. process.name ,that's really bothering.
Any idea about this differences? Appreciate any response!

Please remember to read the discussion listed below first before submitting new issues, the answers there might be helpful!

• Hints about about installing beats #37
• Hints about setting up beats #39

I try to import data use "official ElasticSearch service" , and I encountered the following problem.

what setting I miss?
or I need to change json file by myself to fit the import format?

It means docker is not work?
what can I fix it?

Hello TA,

Just want to make sure something.

1. I want to deploy Elasticsearch in Docker on my machine directly, the VM is way too fat. In order to do so, I want to make sure that
   1. Does the Ubuntu VM only contain the deployed Elasticsearch?
   2. If so, did you add other configuration on it? Can I just docker-compose up the files in the elastic.zip?
2. Since you also provide us a Windows 10 VM, if my host machine is Windows 10 already, am I required to use the Windows 10 VM you provided? (Maybe you've configure something in the Windows 10 VM already?)

Thank you.

After the change of host_id, seems like it didn't work and it still only have 236617..i already change my id in the winlogbeat.yml tho, and actually this number -236617 doesn't even exist in this yml file nor any other file...

Hello,
Does anyone know that how can I logout windows without closing the winlogbeat?
I thought that if I logout windows, the winlogbeat will close, and no log will be transport to ELK stack.

Hi TA, if I want to set up elasticdump, do I need to set up elk which is set up in the previous project, or I just need to install node.js and directly input json file

助教你好:
我有設定 Virtual Box 的 port forwarding，並嘗試在本機瀏覽器打開 Kibana，但顯示 "Kibana server is not ready yet."，請問我該從哪裡檢查環境是不是有出錯?😭

( 底下是我 docker-compose up 後所觀察到的 messages，不確定是不是有沒設定到的地方 )

Hi all,

Does anyone know how to modify field.hostname on kibana? I get "unknown" as default, but TAs' mentioned that it should be set as student id.

Thanks very much for any suggestion.

Kibana connecting problem

• If you have trouble connect to your Kibana.
  Check if you are using Virtual Box. If that's the case, you should survey the port forwarding problem.

Kibana Login account

• The default account is like this below,
  username: admin
password: admin

The output of your beats

• The output should be Logstash, not the ElasticSearch.

Winlogbeat issue

• You should run winlogbeat as admin, otherwise, some events may not be found in kibana.

Screensaver issue

• Your windows should be education version not individual/home.

Notice

• That's correct if you don't find event.code field in packetbeat log. You don't need to contain event.code field for the packetbeat log.
• Google is your friend, try to google error message
• Even if some scenarios you fail to accomplish, you can still try the best to write down the process and your attempts. We will give some points after full consideration.

Hello, is there has a test case without any attack?
I think Test_4 doesn't have any attack.

Hello, TAs.
According to ppt, logs for C&C attack is about "the attacker exfiltrate some files from the victim on an unusual port", but after some investigation with Kibana, I still could not found lots of information with those packetbeat logs. Is there any further information or tips we can get about the attack? Thanks.

I did the step that TA ask me to do(sudo docker system/volume prune , then try sudo docker-compose up again), however even the first 2 successes, it still tells me that I have invalid argument for sudo docker-compose up

Dear TAs,

Here are what work for me so far

• Ubuntu - ELK is up and running (I can log into kibana with admin/admin)
• Windows 10 - Installed winlogbeat and successfully started the service.

The problem

No logs from winlogbeat were sent to elasticsearch (?)

What I've tried

I've read the official documentation of elastic and, if I'm not mistaken, winlogbeat should send the logs via port 9200.

The official doc suggests that the user should run the following command to verify if the nodes are running:

$ curl -X GET "localhost:9200/_cat/nodes?v&pretty"

However, I received this output:

$ curl -X GET "localhost:9200/_cat/nodes?v&pretty"
curl: (52) Empty reply from server

Visiting this URL via a browser gives:

My winlogbeat.yml

Using the following config, I can start the winlogbeat service successfully.

# =================================== Kibana ===================================
setup.kibana:
  # Kibana Host
  host: "192.168.131.137:5601"

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["192.168.131.137:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "admin"
  password: "admin"

The problem is that there isn't any log from winlogbeat showing up in elasticsearch.
Did I miss something? Thanks in advance! 🥺 🥺

I tried this, but "feilds.hostname" attribute in Kibana is still unknown.
What is the problem?

Dear TAs,

as title, as I apply the following settings:

My 'winlogbeat.yml':

setup.kibana:
  host: "192.168.50.89:5601"
  username: "admin"
  password: "admin"

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  hosts: ["192.168.50.89:5044"]

Port forwarding:

I can load into the dashboard successfully in Windows, but I can't find any index that contains 'logstash*' in Kibana and create it as an index pattern.

Did I miss anything? 🥺

Hello, I find Attack_1,3,5 have CVE-2020-0796-RCE-POC AccessList in winlogbeat.
It is mean one of Attack_1,3,5 has Malicious Attachment?
If no repetitive attacks, Why winlogbeat have its accessList log?

Test4 case

• We provided an additional test case for Attack4 in newe3 platform, it has more obvious behavior in it.

Elasticdump problem

• You can refer to #24, they have some possible solutions

Output

• For the output, you only need to output Attack_ID. You don't need to output the attack name.

Format issue

• About the format issue, always follow the spec.

Notice

• Make sure you have read old(closed) issue before you post issues.
• We might directly close the issue if it is a duplicate problem.

/elastic& sudo docker-compose up 打出這行後
sudo: docker-compose: Command not found 回應是這樣

Hi all,

I can receive "login" log if my field.hostname is "unknown" as default. But after I modify the hostname to student id in winlogbeat.yml, I can't receive "login" log but only "user account management". I've tried to restart winlogbeat and all docker containers in ubuntu but still not working.

Does anyone know how to solve this problem? Or is there any other way to modify field.hostname without modifing winlogbeat.yml?

Thanks for any suggestion.

不知道有沒有人跟我遇到一樣的問題

已經好幾個小時卡在這了
winlogbeat就照著官網改，然後改用logstash連線
start-service winlogbeat之後在kibana上怎麼跑都一樣

唯一想到可能的bug是官網說要連至logstash下這條
.\winlogbeat.exe setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["192.168.153.133:9200"]'
可是他只會噴

而且我做完的朋友跟我說不用下 .\winlogbeat setup的指令(?)

PDF上面似乎沒有說明哪個data對應到哪種攻擊，但是這樣我在做test的時候就沒辦法確定我的model判斷的是否準確。
想請問助教能否提供這方面的資訊，謝謝。

照著助教的步驟做，在網頁上填上 your_Ubuntu_ IP:5601. ，上面卻跑出kibana server is not ready yet，這問題該如何解決?
還有kibna是這步驟做完就有結果，就可以做作業了，那為甚麼還需要在windows上裝Winlogbeat 和 Packetbeat ?

I have referred to closed issue #24
But,I have problem,
How to upload the file packetbeat.json to data visualizer with exceeding 100 MB size?
I had google it, but still not found solution.(or maybe no solution?)
Or, I need to use elasticdump to upload file with large size?

thanks.

可以請問助教整個步驟是如何進行的嗎?

第一個步驟下載完our docker files後已經是包含了甚麼文件?
我看裡面有docker-compose.yml 、 internal-use.yml 、logstash.yml 、logstash.conf
(1)而裡面這個docker-compose跟從網路上載下來的有什麼不一樣?
(2)哪ELK Stack又去哪裡了?Elasticsearch是包含docker-compose.yml internal-use.yml 就是Elasticsearch ?Logstash裡面有了就是logstash.yml 、logstash.conf，需要設定什麼開啟甚麼嗎? Kibana是Winlogbeat、Packetbeat嗎?
(3)sudo docker compose up跑完已經是跑了Elasticsearch、Logstash嗎?最後一步驟聯網才是開啟Kibana跑Kibana嗎?
(4)跑Elasticsearch、Logstash是在虛擬機上跑，所以只有Kibana需要處理port問題，也就是5601、5044?如果不是的話這還有少甚麼?

(5) VMnet1為主機ip，enp0s為客機ip?
(6)前面問題說要確認能不能連在windows上打ping 10.0.2.15，不過我的顯示這樣是沒有連到?

As title.
If so, what should we output?(ex. testcase1: sql injection and ddos?)
or maybe they all only have 1 attack scenario?

official docs :
https://www.elastic.co/guide/en/beats/winlogbeat/current/winlogbeat-module-security.html

I think 4802-4803 4667-4668 is essential to this project, but winlogbeat doesn't support it,
I wonder if this is expected,
If yes, I think I need some hints : ( (maybe filebeat((? , I almost tried to modify and recompile winlogbeat ......)

Scenario:
when I try to use "elasticdump" to import log data to my elastic search,I got an error.
error log like below:
{
_index: 'my_index',
_type: '_doc',
_id: '4mwlHnYBWeK1m2KP0ZrY',
status: 400,
error: {
type: 'mapper_parsing_exception',
reason: 'failed to parse',
caused_by: {
type: 'not_x_content_exception',
reason: 'Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes'
}
}
}

In ppt, the submission format is

<student_id>.zip
  |- project_code/
  |- <student_id>.pdf

In spec, the submission format is

<student_id>.zip
  |- <student_id>/
      |- project_code/
      |- report.pdf

Which is the correct submission format?