I am in dilemma because it is not clear to me what do you need to know. Is this about how search engine works or how search engine optimization process works? These two are different. I am trying to explain briefly both topics in my answer.
Search engines are displays the best web sites that are relevant to the search queries by user. The more resourceful the site, the higher its position will be in search engines. Search Engine Optimization (SEO) is a way to show search engines that your website belongs to the first place on the stack. Webmasters uses various techniques to get top positions in search results.
Search engines identify Webpages based on keywords -- important phrases that are strongly related to the content of the page. Computer programs that called spiders or crawler search the web to analyze individual page.
Search engine acquire keywords from various places within a websites such as web page title and headers. However, if you use keywords excessively within a page search engine might treat your page as a spam.
There is another aspect those search engines consider is link analysis. That means the number of websites and their value that linked with that particular webpage. Google gives more importance to that pages which are connected to more authority websites than others are and places them in higher in search results.
You will find many resources on the web that will help you to understand search engine optimization. Some of them are below
Search Engine Optimization (SEO)
Beginners guide to SEO
What is SEO