I want to capture the packet analysis so as to detect intrusion either in a live way(which I think is more difficult...) or by capturing these details and comparing them to any available signature(if any) to find match.
Do you think I will be able to find attack signatures or should I create my own algorithm for example-
It'd be a better idea to use something above like Snort/Suricata just so you can get an idea of it first. It sounds like you have a lot of studying to do around these technologies.
Per above an IDS is useless if it can't detect a live signature as the packet traverses the interface. Having the fastest method of detection (live) is mandatory as you'd rather not know about a certain exploit after it already happened. Also, an
IPS is more of the industry standard currently where the appliance actually drops the traffic it sees as malicious before it ever gets to the server/destination. A simple IDS just alerts you about it but forwards the traffic still.
You would definitely want to reverse engineer some of the traffic fingerprints/signatures to get an idea of what they're looking for and how to write them. Both of the utilities mentioned above are open source so you should be able to extract the signature database (unless someone else has a direct URL).
if I get a too big number of requests on a server-->just say it is a denial of service attack?
How big is too big? What is your plan for avoiding false positives?
You also may be interested in a SIEM OE type of utility or feature that can do correlation rules. Then you can make something like 'alert for traffic that has greater than 6 invalid logins but then eventually a successful login from the same source address' for server access and similar.
OR do you think anomaly based detection would be an easier choice?
Anomaly detection would be much more difficult as you're trying to match around a vague behavior rather than a known malicious piece of data in the packet. The additional challenges mainly stem from eliminating false positives and actually identifying behavior that is potentially malicious.