My Public Notepad: HTTP

Showing posts with label HTTP. Show all posts

Thursday, 5 November 2020

Apache Web Sever on Ubuntu

Many Linux distros, including Ubuntu Debian, come with preinstalled Apache HTTP web server. It is also known as HTTPD (HTTP Daemon).

Its configuration file is /etc/apache2/apache2.conf:

$ cat /etc/apache2/apache2.conf

# This is the main Apache server configuration file. It contains the

# configuration directives that give the server its instructions.

# See http://httpd.apache.org/docs/2.4/ for detailed information about

# the directives and /usr/share/doc/apache2/README.Debian about Debian specific

# hints.

# Summary of how the Apache 2 configuration works in Debian:

# The Apache 2 web server configuration in Debian is quite different to

# upstream's suggested way to configure the web server. This is because Debian's

# default Apache2 installation attempts to make adding and removing modules,

# virtual hosts, and extra configuration directives as flexible as possible, in

# order to make automating the changes and administering the server as easy as

# possible.

# It is split into several files forming the configuration hierarchy outlined

# below, all located in the /etc/apache2/ directory:

# /etc/apache2/

# |-- apache2.conf

# | `-- ports.conf

# |-- mods-enabled

# | |-- *.load

# | `-- *.conf

# |-- conf-enabled

# | `-- *.conf

# `-- sites-enabled

# `-- *.conf

# * apache2.conf is the main configuration file (this file). It puts the pieces

# together by including all remaining configuration files when starting up the

# web server.

# * ports.conf is always included from the main configuration file. It is

# supposed to determine listening ports for incoming connections which can be

# customized anytime.

# * Configuration files in the mods-enabled/, conf-enabled/ and sites-enabled/

# directories contain particular configuration snippets which manage modules,

# global configuration fragments, or virtual host configurations,

# respectively.

# They are activated by symlinking available configuration files from their

# respective *-available/ counterparts. These should be managed by using our

# helpers a2enmod/a2dismod, a2ensite/a2dissite and a2enconf/a2disconf. See

# their respective man pages for detailed information.

# * The binary is called apache2. Due to the use of environment variables, in

# the default configuration, apache2 needs to be started/stopped with

# /etc/init.d/apache2 or apache2ctl. Calling /usr/bin/apache2 directly will not

# work with the default configuration.

# Global configuration

# ServerRoot: The top of the directory tree under which the server's

# configuration, error, and log files are kept.

# NOTE! If you intend to place this on an NFS (or otherwise network)

# mounted filesystem then please read the Mutex documentation (available

# at <URL:http://httpd.apache.org/docs/2.4/mod/core.html#mutex>);

# you will save yourself a lot of trouble.

# Do NOT add a slash at the end of the directory path.

#ServerRoot "/etc/apache2"

# The accept serialization lock file MUST BE STORED ON A LOCAL DISK.

#Mutex file:${APACHE_LOCK_DIR} default

# The directory where shm and other runtime files will be stored.

DefaultRuntimeDir ${APACHE_RUN_DIR}

# PidFile: The file in which the server should record its process

# identification number when it starts.

# This needs to be set in /etc/apache2/envvars

PidFile ${APACHE_PID_FILE}

# Timeout: The number of seconds before receives and sends time out.

Timeout 300

# KeepAlive: Whether or not to allow persistent connections (more than

# one request per connection). Set to "Off" to deactivate.

KeepAlive On

# MaxKeepAliveRequests: The maximum number of requests to allow

# during a persistent connection. Set to 0 to allow an unlimited amount.

# We recommend you leave this number high, for maximum performance.

MaxKeepAliveRequests 100

# KeepAliveTimeout: Number of seconds to wait for the next request from the

# same client on the same connection.

KeepAliveTimeout 5

# These need to be set in /etc/apache2/envvars

User ${APACHE_RUN_USER}

Group ${APACHE_RUN_GROUP}

# HostnameLookups: Log the names of clients or just their IP addresses

# e.g., www.apache.org (on) or 204.62.129.132 (off).

# The default is off because it'd be overall better for the net if people

# had to knowingly turn this feature on, since enabling it means that

# each client request will result in AT LEAST one lookup request to the

# nameserver.

HostnameLookups Off

# ErrorLog: The location of the error log file.

# If you do not specify an ErrorLog directive within a <VirtualHost>

# container, error messages relating to that virtual host will be

# logged here. If you *do* define an error logfile for a <VirtualHost>

# container, that host's errors will be logged there and not here.

ErrorLog ${APACHE_LOG_DIR}/error.log

# LogLevel: Control the severity of messages logged to the error_log.

# Available values: trace8, ..., trace1, debug, info, notice, warn,

# error, crit, alert, emerg.

# It is also possible to configure the log level for particular modules, e.g.

# "LogLevel info ssl:warn"

LogLevel warn

# Include module configuration:

IncludeOptional mods-enabled/*.load

IncludeOptional mods-enabled/*.conf

# Include list of ports to listen on

Include ports.conf

# Sets the default security model of the Apache2 HTTPD server. It does

# not allow access to the root filesystem outside of /usr/share and /var/www.

# The former is used by web applications packaged in Debian,

# the latter may be used for local directories served by the web server. If

# your system is serving content from a sub-directory in /srv you must allow

# access here, or in any related virtual host.

Options FollowSymLinks

AllowOverride None

Require all denied

</Directory>

AllowOverride None

Require all granted

</Directory>

Options Indexes FollowSymLinks

AllowOverride None

Require all granted

</Directory>

#<Directory /srv/>

# Options Indexes FollowSymLinks

# AllowOverride None

# Require all granted

#</Directory>

# AccessFileName: The name of the file to look for in each directory

# for additional configuration directives. See also the AllowOverride

# directive.

AccessFileName .htaccess

# The following lines prevent .htaccess and .htpasswd files from being

# viewed by Web clients.

Require all denied

</FilesMatch>

# The following directives define some format nicknames for use with

# a CustomLog directive.

# These deviate from the Common Log Format definitions in that they use %O

# (the actual bytes sent including headers) instead of %b (the size of the

# requested file), because the latter makes it impossible to detect partial

# requests.

# Note that the use of %{X-Forwarded-For}i instead of %h is not recommended.

# Use mod_remoteip instead.

LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined

LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined

LogFormat "%h %l %u %t \"%r\" %>s %O" common

LogFormat "%{Referer}i -> %U" referer

LogFormat "%{User-agent}i" agent

# Include of directories ignores editors' and dpkg's backup files,

# see README.Debian for details.

# Include generic snippets of statements

IncludeOptional conf-enabled/*.conf

# Include the virtual host configurations:

IncludeOptional sites-enabled/*.conf

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

Let's try to find the index page of the default web site. From the config file above, we can see where to look for website content:

$ ls /etc/apache2/sites-

sites-available/ sites-enabled/

$ ls /etc/apache2/sites-available/

000-default.conf default-ssl.conf

$ ls /etc/apache2/sites-enabled/

000-default.conf

$ cat /etc/apache2/sites-enabled/000-default.conf

# The ServerName directive sets the request scheme, hostname and port that

# the server uses to identify itself. This is used when creating

# redirection URLs. In the context of virtual hosts, the ServerName

# specifies what hostname must appear in the request's Host: header to

# match this virtual host. For the default virtual host (this file) this

# value is not decisive as it is used as a last resort host regardless.

# However, you must set it for any further virtual host explicitly.

#ServerName www.example.com

ServerAdmin webmaster@localhost

DocumentRoot /var/www/html

# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,

# error, crit, alert, emerg.

# It is also possible to configure the loglevel for particular

# modules, e.g.

#LogLevel info ssl:warn

ErrorLog ${APACHE_LOG_DIR}/error.log

CustomLog ${APACHE_LOG_DIR}/access.log combined

# For most configuration files from conf-available/, which are

# enabled or disabled at a global level, it is possible to

# include a line for only one particular virtual host. For example the

# following line enables the CGI configuration for this host only

# after it has been globally disabled with "a2disconf".

#Include conf-available/serve-cgi-bin.conf

</VirtualHost>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

And now we can see where is index.html:

$ ls /var/www/html/

index.html

We can see the content of the index.html if we type in /var/www/html/index.html or http://localhost in the browser:

To check which processes are currently listening on which ports:

$ sudo lsof -i -P -n | grep LISTEN

...

apache2 1456 root 4u IPv6 31674 0t0 TCP *:80 (LISTEN)

apache2 10763 www-data 4u IPv6 31674 0t0 TCP *:80 (LISTEN)

apache2 10764 www-data 4u IPv6 31674 0t0 TCP *:80 (LISTEN)

apache2 10765 www-data 4u IPv6 31674 0t0 TCP *:80 (LISTEN)

apache2 10766 www-data 4u IPv6 31674 0t0 TCP *:80 (LISTEN)

apache2 10767 www-data 4u IPv6 31674 0t0 TCP *:80 (LISTEN)

...

To check if Apache2 service is running:

$ sudo systemctl status apache2

[sudo] password for xxx:

● apache2.service - The Apache HTTP Server

Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)

Drop-In: /lib/systemd/system/apache2.service.d

└─apache2-systemd.conf

Active: active (running) since Fri 2020-09-25 10:49:52 BST; 6 days ago

Process: 4039 ExecReload=/usr/sbin/apachectl graceful (code=exited, status=0/SUCCESS)

Main PID: 1456 (apache2)

Tasks: 6 (limit: 4915)

CGroup: /system.slice/apache2.service

├─1456 /usr/sbin/apache2 -k start

├─4043 /usr/sbin/apache2 -k start

├─4044 /usr/sbin/apache2 -k start

├─4045 /usr/sbin/apache2 -k start

├─4046 /usr/sbin/apache2 -k start

└─4047 /usr/sbin/apache2 -k start

Sep 29 07:08:29 bobox systemd[1]: Reloaded The Apache HTTP Server.

Sep 30 00:08:52 bobox systemd[1]: Reloading The Apache HTTP Server.

Sep 30 00:08:52 bobox apachectl[17514]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the '

Sep 30 00:08:52 bobox systemd[1]: Reloaded The Apache HTTP Server.

Oct 01 00:06:42 bobox systemd[1]: Reloading The Apache HTTP Server.

Oct 01 00:06:42 bobox apachectl[12396]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the '

Oct 01 00:06:42 bobox systemd[1]: Reloaded The Apache HTTP Server.

Oct 02 00:06:29 bobox systemd[1]: Reloading The Apache HTTP Server.

Oct 02 00:06:29 bobox apachectl[4039]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'S

Oct 02 00:06:29 bobox systemd[1]: Reloaded The Apache HTTP Server.

How to check if Apache service is enabled at boot time?

$ sudo systemctl is-enabled apache2

enabled

How to disable Apache service at boot time:

$ sudo systemctl disable apache2

Synchronizing state of apache2.service with SysV service script with /lib/systemd/systemd-sysv-install.

Executing: /lib/systemd/systemd-sysv-install disable apache2

The previous command does not stop the service which is already running:

$ sudo lsof -i -P -n | grep LISTEN

...

apache2 1456 root 4u IPv6 31674 0t0 TCP *:80 (LISTEN)

apache2 4043 www-data 4u IPv6 31674 0t0 TCP *:80 (LISTEN)

apache2 4044 www-data 4u IPv6 31674 0t0 TCP *:80 (LISTEN)

apache2 4045 www-data 4u IPv6 31674 0t0 TCP *:80 (LISTEN)

apache2 4046 www-data 4u IPv6 31674 0t0 TCP *:80 (LISTEN)

apache2 4047 www-data 4u IPv6 31674 0t0 TCP *:80 (LISTEN)

...

To stop currently running Apache service:

$ sudo systemctl stop apache2

To verify that the service is stopped:

$ sudo systemctl status apache2

● apache2.service - The Apache HTTP Server

Loaded: loaded (/lib/systemd/system/apache2.service; disabled; vendor preset: enabled)

Drop-In: /lib/systemd/system/apache2.service.d

└─apache2-systemd.conf

Active: inactive (dead)

Sep 30 00:08:52 bobox systemd[1]: Reloaded The Apache HTTP Server.

Oct 01 00:06:42 bobox systemd[1]: Reloading The Apache HTTP Server.

Oct 01 00:06:42 bobox apachectl[12396]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the '

Oct 01 00:06:42 bobox systemd[1]: Reloaded The Apache HTTP Server.

Oct 02 00:06:29 bobox systemd[1]: Reloading The Apache HTTP Server.

Oct 02 00:06:29 bobox apachectl[4039]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'S

Oct 02 00:06:29 bobox systemd[1]: Reloaded The Apache HTTP Server.

Oct 02 04:36:27 bobox systemd[1]: Stopping The Apache HTTP Server...

Oct 02 04:36:27 bobox apachectl[26754]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the '

Oct 02 04:36:27 bobox systemd[1]: Stopped The Apache HTTP Server.

We can also verify that it's not listening on any ports:

$ sudo lsof -i -P -n | grep LISTEN

// shows no apache2 entries

Thursday, 18 April 2019

Http Client in Node.js

Making HTTP(S) calls from the application is very common requirement. Node.JS world offers lots of options about how to implement it, or better to say - which package to use.

Native solution

Node.JS comes with native http and https packages.

http.get example

Do a Basic HTTP Request with Node.js

Request package

Request - Simplified HTTP client

It does not support Promises so if we need to wrap it into a Promise ourselves...something like:

/**

* Function which fetches http response asynchronously.

* It is implemented as a simple Promise wrapper around request API.

* @param url Request URL.

function fetchData(url: string): Promise<string> {

return new Promise((resolve, reject) => {

if (!url) {

reject(ErrorFactory.createInvalidArgumentError("url"));

}

request(url, (error: any, response: request.Response, body: any) => {

if (error) {

reject(error);

} else {

if (response) {

console.log("statusCode: ", response.statusCode);

if (body) {

resolve(body);

} else {

reject(ErrorFactory.createFalsyObjectError("body"));

}

} else {

reject(ErrorFactory.createFalsyObjectError("response"));

}

});

}

Request package with native promises

Request-Promise-Native

node-fetch package

window.fetch API for Node.js

node-fetch

Axios package

Promise based HTTP client for the browser and node.js

axios

Some References

fetch vs axios vs request-promise vs superagent

Why I won’t be using Fetch API in my apps

Why do I stop using Fetch? (and what are the alternative options?)

4 + 1 ways for making HTTP requests with Node.js: async/await edition

Conclusion

I've been looking solutions which support Promises. node-fetch seems to be traditional approach and Axios seems to be very popular among developers now.

Wednesday, 5 August 2015

How to sniff HTTP traffic on the local Wi-Fi network in 10 steps

We need:

(1) Attacker: Linux machine with two Wi-Fi cards; I am using Kali with internal Atheros and external Alfa (AWUS036NH) WiFi card.
(2) Victim: mobile device; I am using smartphone
(3) Wi-Fi router with set up Wi-Fi network

Steps:

(1) Verify that both Wi-Fi network cards are connected to the same Wi-Fi network:

root@kali:/# iwconfig
wlan1 IEEE 802.11bgn ESSID:"MYWIFINET"
Mode:Managed Frequency:2.457 GHz Access Point: 10:AD:AF:CD:A7:A4
Bit Rate=1 Mb/s Tx-Power=20 dBm
Retry short limit:7 RTS thr:off Fragment thr:off
Encryption key:off
Power Management:off
Link Quality=70/70 Signal level=-37 dBm
Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0
Tx excessive retries:0 Invalid misc:4 Missed beacon:0

eth0 no wireless extensions.

lo no wireless extensions.

wlan0 IEEE 802.11bgn ESSID:"MYWIFINET"
Mode:Managed Frequency:2.457 GHz Access Point: 10:AD:AF:CD:A7:A4
Bit Rate=65 Mb/s Tx-Power=16 dBm
Retry short limit:7 RTS thr:off Fragment thr:off
Encryption key:off
Power Management:off
Link Quality=64/70 Signal level=-46 dBm
Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0
Tx excessive retries:2 Invalid misc:332 Missed beacon:0

Atheros is wlan0 and Alpha is wlan1:

root@kali:/# ifconfig
eth0 Link encap:Ethernet
...

lo Link encap:Local Loopback
...

wlan0 Link encap:Ethernet HWaddr ac:ba:ad:aa:aa:aa
inet addr:192.168.0.3 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::9eb7:dff:fe04:d2f5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:993402 errors:0 dropped:16671 overruns:0 frame:0
TX packets:1037777 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1231583696 (1.1 GiB) TX bytes:293024209 (279.4 MiB)

wlan1 Link encap:Ethernet HWaddr 00:c0:ca:bb:bb:bb
inet addr:192.168.0.9 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::2c0:caff:fe59:23d0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:14 errors:0 dropped:0 overruns:0 frame:0
TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1164 (1.1 KiB) TX bytes:1882 (1.8 KiB)

(2) Put one of Wi-Fi interfaces into monitor mode:

root@kali:/# airmon-ng start wlan1

Found 5 processes that could cause trouble.
If airodump-ng, aireplay-ng or airtun-ng stops working after
a short period of time, you may want to kill (some of) them!
-e
PID Name
2539 NetworkManager
2644 wpa_supplicant
3037 dhclient
19213 dhclient
20374 dhclient
Process with PID 20374 (dhclient) is running on interface wlan1
Process with PID 19213 (dhclient) is running on interface wlan0

Interface Chipset Driver

wlan1 Ralink RT2870/3070 rt2800usb - [phy4]
(monitor mode enabled on mon0)
wlan0 Atheros AR9485 ath9k - [phy0]

(3) Go to Wireshark's WPA PSK (Raw Key) Generator page: https://www.wireshark.org/tools/wpa-psk.html
Type in your Wi-Fi network's name and password and click on Generate PSK button.

(4) Start Wireshark. If it is not installed, install it with apt-get install wireshark command.

(5) In Wireshark: go to Capture --> Options and check "Use promiscuous mode on all interfaces"

(6) In Wireshark: go to Edit --> Preferences --> Protocols --> IEEE802.11, check "Enable decryption" option and add generated PSK key as new wpa-psk key in Decryption Keys.

(7) In Wireshark's main dashboard select monitor interface created by airmon-ng; that is mon0 in my case.
Press "Start" button in order to start live capture.

(8) Connect mobile device to Wi-Fi network. Wireshark has to capture handshake packets exchanged between the victim and the router when victim joins Wi-Fi network.

(9) In the browser of the victim's device type in any http address and allow it to load. I typed http://m.bbc.co.uk/weather/2643743 in order to get weather forecast for London from BBC Weather mobile webiste.

(10) Stop Wireshark and search for the HTTP traffic which goes between any IP address which is not the IP address of local Wi-Fi interfaces. In my case that was 192.168.0.5. I could see DNS requests to all services my smartphone uses (Google, Facebook, Whatsapp...) and also DNS query for m.bbc.co.uk, and HTTP GET request that was sent!

Thursday, 15 March 2012

How to sniff SOAP messages exchanged between WCF Service Host and Test Client

In my article "How to create and test WCF Web Service" I described how to implement simple Calculator service and test it from standalone WCF Service Host and Test Client.

SOAP Web Service and client exchange SOAP messages (request and response) during web method calls. SOAP messages are wrapped with HTTP messages and could be viewed and analysed by using some HTTP sniffer application.

For HTTP (SOAP) debugging I usually use Fiddler. It is a HTTP Proxy running on port 8888 on the local host. By default it intercepts HTTP traffic between web browsers and servers but can be set to sniff HTTP packets for any application that accepts HTTP proxies.

If we set Calculator web service to listen on port 8733 (defined in baseAddress URL in App.config) we can run WCF Service Host:

C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE>WcfSvcHost.exe /service:"c:\DEVELOPMENT\RESEARCH\C#\WCF\Web Services\CalculatorServiceLibrary\CalculatorServiceLibrary\bin\Debug\CalculatorServiceLibrary.dll" /config:"c:\DEVELOPMENT\RESEARCH\C#\WCF\Web Services\CalculatorServiceLibrary\CalculatorServiceLibrary\bin\Debug\CalculatorServiceLibrary.dll.config

...and Test Client:

C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE>WcfTestClient.exe http://localhost:8733/Design_Time_Addresses/CalculatorServiceLibrary/Service1/mex

Their model of communication is simple: Web Service Host listens on port 8733 for incoming SOAP requests (on service endpoint with URL http://localhost:8733/Design_Time_Addresses/CalculatorServiceLibrary/Service1) or metadata requests (on metadata endpoint with URL http://localhost:8733/Design_Time_Addresses/CalculatorServiceLibrary/Service1/mex). Once client sends request, host is loading Web Service dll, executing web method, packing result into SOAP response and sending it back within HTTP response:

Fiddler is by default listening on port 8888 and in this constellation cannot intercept traffic between WCF Host and Client. We need to set it as a HTTP proxy which will forward all HTTP traffic to port where WCF Host is listening - port 8733. This can be achieved by adding ReverseProxyForPort DWORD value to HKCU\SOFTWARE\Microsoft\Fiddler2 and setting it to 8733. Fiddler must be restarted to fetch this change.

Client can send now HTTP requests to port 8888 in which case Fiddler will be able to intercept and display HTTP messages. It will forward them to port 8733 so they will reach Web Service Host. Of course, client can still send HTTP requests directly to port 8733, bypassing Fiddler.

When we run WCF Test Client we are providing it with metadata URL. This metadata contains service URL which is one defined in the service's App.config. This URL contains port number and it is set to 8733. Client will use this URL in order to make web service call. But how can we trick client so it uses port 8888 instead of 8733? When client gets metadata, it stores it in its own config file - client.dll.config. This file is stored in user's temporary folder, e.g. in c:\Users\Bojan\AppData\Local\Temp\Test Client Projects\10.0\636e9680-a905-4c50-a9ed-99562eb36701\ and looks something like this:

client.dll.config:

We can modify this config file before invoking web methods. We can do it manually through any text editor or simply from Test Client application. We can navigate client to target port 8888 now (make sure Fiddler is running - it listens on port 8888):

C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE>WcfTestClient.exe http://localhost:8888/Design_Time_Addresses/CalculatorServiceLibrary/Service1/mex

WCF Test Client: