Results 1 to 12 of 12

Thread: Spanish accents & MySQL select query

  1. #1
    Join Date
    Jun 2007

    Question Spanish accents & MySQL select query


    I have struggled for months with this one. I originally placed all of the Spanish into the database with the accent characters in HTML-entity form.

    For example:
    ¡Me encanta la fotografía!
    This would correctly display the characters on screen, but the word "fotografía" could not be matched in a select query (with the possible exception of actually typing the HTML-entity).

    I wanted everything to be in UTF8 format, so I then tried converting the entire database to this, replacing all of the HTML-entities with their UTF8 equivalents. Now I can search, and find, the Spanish accented words, but ONLY if the accents are entered into the query.

    For example, if the word is "Jesús", the un-accented form "Jesus" will not match, but "Jesús" will.

    I have heard there is a way to ignore accents, similar to ignoring case.

    How can I properly set this up?

    I _VERY MUCH_ appreciate any helpful hints on this.



  2. #2
    Join Date
    Aug 2005
    The Garden State
    how are you searching? full text? or just IS or LIKE ? if you're not doing FT then try that first.
    Acceptable Use | SQL Forum FAQ | celery is tasteless | twitter

    celery is tasteless - currently needing some UI time

  3. #3
    Join Date
    Jun 2007
    I've been using queries, via my perl script, of this form:

    SELECT * from Database.Table WHERE TextColumn LIKE "%keyword%" LIMIT 0, 10;
    Part of my issue here is that the script needs to handle multiple languages. I have multiple tables in the database, one for each language: English, Spanish, Chinese, Thai, etc. Unfortunately, I don't think I can use a FULLTEXT search on the Asian languages. Also, in order to handle multiple languages equally, the database is setup to do everything in UTF8.



  4. #4
    Join Date
    Aug 2005
    The Garden State
    Quote Originally Posted by PolyGreat View Post
    I don't think I can use a FULLTEXT search on the Asian languages
    Of course you can. Try it.

    Of course, it's unlikely that a simplified chinese text can ever match against english. not even the google can do that.
    Last edited by chazzy; 10-28-2008 at 07:27 PM.
    Acceptable Use | SQL Forum FAQ | celery is tasteless | twitter

    celery is tasteless - currently needing some UI time

  5. #5
    Join Date
    Jun 2007
    So, is this accent-insensitive search only going to be available when using fulltext? Is there a way of making it work with an ordinary LIKE statement, or perhaps with RLIKE?

    Spanish is the only database I'm having issues with right now, and I'm hesitant to make major changes to the search queries when all of the other languages are operating smoothly.

    As I understand, using a FULLTEXT search is not all that similar to doing a LIKE search in several aspects: 1) Ordering of results, 2) Wordlengths, and 3) System requirements.



  6. #6
    Join Date
    Jun 2007
    I'm still looking for an answer to this. I've asked a few of my expert friends, who said they'd get back to me, but never did...so I presume they did not find the answer either.

    There are a few other reasons why I am not doing a fulltext search, and thus cannot consider it as a "best option" in my case. Is there no way to do an accent-independent search in MySQL without fulltext?

    Is this really that difficult?



  7. #7
    Join Date
    Aug 2005
    The Garden State
    as far as I am concerned, I've already provided you a viable solution. it is your choice whether to try it not.
    Acceptable Use | SQL Forum FAQ | celery is tasteless | twitter

    celery is tasteless - currently needing some UI time

  8. #8
    Join Date
    May 2009

    Case Insensitive, Accent Insensitive


    Most of the databases today have the "Collation" option in which you define your database collation in certain "language oriented", Case Sensitive/Insensitive and Accent Sensitive/Insensitive.

    Defining your database (all), or only a table with a desired collation will allow you to query data without having to deal with accents or case. Example:

    My database is MS SQL 2005 and the collation of my database is defined as:


    This definition is made at the time of database creation, or later your can change properties of database or only a table with an specific collation.

    Latin1_General_CI_AI collation gives you the chance to have latin caracters and include most of the latin and european languages (excluding japanese, russian and some others with very funky characters).
    CI means Case Insensitive and AI means Accents Insensitive

    When creating a database or changing the collation you are given multiple options of Latin or other character sets.
    To change the collation of a table, set Extended Properties of the table and change the collation.

    In case of MS SQL, ther is no way to define specifically UTF-8 set since database is multi-character capable (that says Microsoft). However, if you store the data from a page defined as utf-8, data will be saved into table as UTF-8 and when retrieved will be utf-8 as well. This is a tricky thing since data can be utf-8 but retrieved in a page with a different charset, SQL will convert.

    FYI: ASP language on IIS does not handle utf-8 well, sometimes display data as charset=iso-8859-1 or so, so funny chars are shown. MS says utf-8 is not supported. You can use it but half of the time will sitch data to funny chars.
    ASPX is other stoy, is fully functional with UTF-8 as well as php and other languages.

    If you use Postgre or MYSQL database, both have this collation types witha case and accent sensitive/insensitive and full definition for utf-8.

    In short, defining collation with AI and CI collation, you only need to query your database and accents and case will not be taken in consideration, this means user can type something with or without accent. Example:

    User type: ídea and results come with records for: Idea, idea, ídea

  9. #9
    Join Date
    Jun 2007
    Are you sure that MySQL supports these *_AI character sets?

    To be frankly honest, I'm taking a hard look at non-MySQL solutions right now, because MySQL is behind the times on the basics like this. Another feature I need that it lacks is that of word proximity searching.

    I'm just tired of dealing with this. I have only one possible solution, it appears, with MySQL: create a column for searching in which ALL of the accents have been replaced with non-accented characters, and then return the search results from a separate column where the original text has been preserved. But that means double the data space, and more mess to deal with.

    Thank you for taking the time to share the possibility. I just haven't seen it be a reality for MySQL.



  10. #10
    Join Date
    Jul 2009
    try this

    $search = str_replace('%', '[ÑñÇçÀÁÂÃÄÅÈÉÊËÌÍ&#20 6;ÏÒÓÔÕÖÙÚÛÜàáâãäå&# 232;éêëìíîïòóôõöùúûü]', $search);

  11. #11
    Join Date
    Jul 2009
    or this
    $search = str_replace('%', '[aàáâãäå]', $search);
    $search = str_replace('%', '[eèéêë]', $search);
    $search = str_replace('%', '[iìíîï]', $search);
    $search = str_replace('%', '[oòóôõö]', $search);
    $search = str_replace('%', '[uùúûü]', $search);
    $search = str_replace('%', '[AÀÁÂÃÄÅ]', $search);
    $search = str_replace('%', '[EÈÉÊË]', $search);
    $search = str_replace('%', '[IÌÍÎÏ]', $search);
    $search = str_replace('%', '[OÒÓÔÕÖ]', $search);
    $search = str_replace('%', '[UÙÚÛÜ]', $search);

  12. #12
    Join Date
    Sep 2009

    Smile Solution

    SELECT *
    FROM `table`
    WHERE `column` LIKE _utf8 '%word with accent%' COLLATE utf8_unicode_ci;

    That will work.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
HTML5 Development Center



X vBulletin 4.2.2 Debug Information

  • Page Generation 0.13860 seconds
  • Memory Usage 2,974KB
  • Queries Executed 15 (?)
More Information
Template Usage (34):
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_global_above_footer
  • (1)ad_global_below_navbar
  • (1)ad_global_header1
  • (1)ad_global_header2
  • (1)ad_navbar_below
  • (1)ad_showthread_firstpost_sig
  • (1)ad_showthread_firstpost_start
  • (1)ad_thread_first_post_content
  • (1)ad_thread_last_post_content
  • (2)bbcode_code
  • (1)bbcode_quote
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)headinclude_bottom
  • (12)memberaction_dropdown
  • (1)navbar
  • (4)navbar_link
  • (1)navbar_moderation
  • (1)navbar_noticebit
  • (1)navbar_tabs
  • (2)option
  • (12)postbit
  • (12)postbit_onlinestatus
  • (12)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 

Phrase Groups Available (6):
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files (26):
  • ./showthread.php
  • ./global.php
  • ./includes/class_bootstrap.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/functions_navigation.php
  • ./includes/class_friendly_url.php
  • ./includes/class_hook.php
  • ./includes/class_bootstrap_framework.php
  • ./vb/vb.php
  • ./vb/phrase.php
  • ./includes/functions_facebook.php
  • ./includes/functions_calendar.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_notice.php
  • ./packages/vbattach/attach.php
  • ./vb/types.php
  • ./vb/cache.php
  • ./vb/cache/db.php
  • ./vb/cache/observer/db.php
  • ./vb/cache/observer.php 

Hooks Called (73):
  • init_startup
  • friendlyurl_resolve_class
  • init_startup_session_setup_start
  • database_pre_fetch_array
  • database_post_fetch_array
  • init_startup_session_setup_complete
  • global_bootstrap_init_start
  • global_bootstrap_init_complete
  • cache_permissions
  • fetch_postinfo_query
  • fetch_postinfo
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • load_show_variables
  • load_forum_show_variables
  • global_state_check
  • global_bootstrap_complete
  • global_start
  • style_fetch
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • strip_bbcode
  • friendlyurl_clean_fragment
  • friendlyurl_geturl
  • forumjump
  • cache_templates
  • cache_templates_process
  • template_register_var
  • template_render_output
  • fetch_template_start
  • fetch_template_complete
  • parse_templates
  • fetch_musername
  • notices_check_start
  • notices_noticebit
  • process_templates_complete
  • friendlyurl_redirect_canonical
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • memberaction_dropdown
  • tag_fetchbit
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • build_navigation_data
  • build_navigation_array
  • check_navigation_permission
  • process_navigation_links_start
  • process_navigation_links_complete
  • set_navigation_menu_element
  • build_navigation_menudata
  • build_navigation_listdata
  • build_navigation_list
  • set_navigation_tab_main
  • set_navigation_tab_fallback
  • navigation_tab_complete
  • fb_like_button
  • showthread_complete
  • page_templates